Glad you're looking at this, Abi. This always seemed to be a strange fear for people to be constantly mentioning. It presupposes that the pre-AI bottleneck for mass bioweapon deployment was the molecular design, which just never passed the smell test.
Thanks for this extremely thoughtful response! I'll jump in with a quick reply because Abi is mostly offline for the next few days, and then I'll let Abi respond in detail.
The key idea that emerged from Abi's research is that there are bottlenecks which do *not* seem likely to be significantly eased, at least in the near term, by current developments. (You cite some of the most important, such as cloud labs and DNA synthesis providers.) Abi based this work on discussions with multiple people from the AI / biosecurity community (some of whom are named in the acknowledgement at the end of the post), including feedback on the final draft before publication. The next couple of installments will go into more detail regarding those bottlenecks, and we'd love further feedback.
You also mention the emergence of contract research organizations (labs-for-hire). This isn't a topic I know much about, and I am not certain whether it came up in discussions, so I'll have to wait for Abi here. I'll also defer to Abi to comment on precisely how her findings relate to the Virology Capabilities Test results; I know this was on her radar, and again the upcoming installments will shed more light here.
Thanks Steve! Really appreciate the engagement, and I want to reemphasize that Abi's project is genuinely a valuable contribution; the practitioner perspective has been missing from this conversation, and I'm glad she's been doing interviews and discussions with people in the field!
I'm interested to see the next installments get concrete on specific bottlenecks that remain, but I think my concern might be less about any single development and more about compound effects of being able to route around *any* of the bottlenecks that used to serve as barriers to bioweapons development.
Also glad CROs are on the radar. Looking forward to the rest of the series!
Your point 1 rests on assumptions that part 3 addresses at length. If you send me your email via SubstackDMs (or however you're comfortable). I'd be happy to send for your thoughts and feedback, as well as the other parts! :)
I'd love to know more about which viruses you envision in the airport example. A more specific example would be helpful because, depending on what type of virus you're proposing, the core bottleneck is different.
An existing virus faces the same frictions it always has: symptomatic spread gets detected fast, and killing people quickly limits transmission. It'd be very expensive way to kill a dozens or a hundred as you note.
An engineered novel virus raises a different problem: we don't actually know whether engineering a pathogen to behave exactly as intended is possible outside of a massive, multi-institutional team doing years of physical testing on humans. To career biologists, it sounds like "anyone can build nuclear weapons because the blueprints are online." It's technically true in a narrow sense, but not really in the physical world.
The biology field has an entirely different, maybe more difficult, nature than building a nuclear weapon or a bioweapon: you don't know whether the end goal you're aiming for (the theoretical virus) is viable, and if it is, whether it works as you expect with any part of the immune system or human biology. These three different problems make the type of physical experiments you need very very large, potentially infinite. The problem is hard because you don't know whether the endpoint is achievable, so you have no way to know if you're really much closer to a novel pathogen.
Aum Shinrikyo had a billion dollars and multidisciplinary experts and still couldn't produce a bioweapon. They had to turn to a chemical weapon.
We do actually approach cybersecurity this way. Lots of things are legal because "common actors" don't use it. Salad bars are a useful vector for bioattacks (see Oregon e.coli attacks) but they're still legal because it's so rare for people to want to do it. Even in cybersecurity, practitioners note that there are a ton of easy-to-exploit vulnerabilities, but people don't exploit them because 1. money laundering is annoying and human-work-intensive, and 2. people with hacking skills tend toward more lucrative cyber-related work, like working with state actors, for example. See Meta's former cyber person Joshua Saxe explain the actor model in cyber: https://joshuasaxe181906.substack.com/p/exploits-dont-cause-cyberattacks.
For context, I'm explaining why much of the biosecurity world is focused on institutional actors, not "groups of laypeople" or "lone terrorists". My post aims to clarify *why* biosecurity folks are focused on the most capable attackers. A lot of headlines give the impression that "any day now, any person can be a bioterrorist".
On the new technologies section:
Cloud labs — can you be more specific about the concern? If dealing with a known virus, how would they hide from the cloud lab that they're working with a known virus? A cloud lab does the highly personalized work of figuring out "how to boot up a virus" process without asking why someone is making a virus? If it's not a known virus, getting from DNA assembly to a functional pathogen requires potentially infinite iterations if you're dealing with anything novel. When dealing with a novel pathogen, how would they know which viruses are compatible with life (not human life)? If you cross that very high obstacle, you'd need many willing humans to figure out which viruses (if any are viable with biology) are viable with human biologies and immune systems.
DNA synthesis — this touches only one part of the bioweapon creation process, and not the hardest part. It gets a lot of attention because it's where AI intersects, but that's not the same as it being the key barrier.
CROs — agreed regulatory oversight matters here (and in cloud labs), and I don't think anyone is arguing against it.
On the 94% virology expert survey: I was glad to see the focus on lab work, but the questions may not capture what actually makes virologists dangerous or not. Multiple biologists told me virology is more like surgery than other STEM fields — the expertise is organism-specific, batch-specific, built up over decades of slow, hands-on learning. The survey gets at a sliver of laboratory protocols that are written down and some tacit knowledge. But the tacit knowledge required for each specific organism is huge and it's unclear whether the test gets at even 5% of it (random number, low confidence, could be <1% could be 15%). The tacit knowledge is the reason why the only person who succeeded at anthrax was a world expert on anthrax.
To be clear, I agree with your advocacy for more biosecurity efforts. But I believe your conclusion has unintended consequences. You said "arguments that downplay AI biosecurity risk— even unintentionally— risk contributing to policy inaction on a problem where the cost of delay could be extremely high."
My response on that: If the AI biosecurity risk is lower, but advocates overstate the risk, then it'll be harder to know when advocates are not crying wolf. It's also not good for group epistemics to see counterarguments as risky. It's a disservice to the field of AI-biosecurity. To get stronger arguments and be someone who governments reach out to, it's pretty core to understand why practitioners downplay it. It's the same reason the government will downplay it. Keeping "downplaying" types of evidence out of public discourse just harms the epistemics of people outside of government and makes it less likely that advocates will affect people inside of government.
Can't wait for your thoughts on the next parts! Do send me your email so I can send the Google Docs. :)
We've been thinking about this a lot at Radvac, the biosecurity nonprofit where I work. I personally think that AI-enhanced biorisk is the largest existential risk we currently face, and will likely be the dominant risk over the next decade. I also think it is very neglected, especially compared to AI safety, which has had an influx of very large numbers of highly skilled people in recent years.
I am mainly worried about actors that do want to kill everyone, including themselves. Emil Torres called these "omnicidal agents". I noticed your piece doesn't address this risk at all. Omnicidal agents are not a theoretical possibility, we already have examples of environmentalists who think that humans are a "cancer" on the planet and are in favor of human extinction. There are also negative utilitarians who are pro-extinction. It's very easy to imagine a mentally-destabilized environmental extremist or negative utilitarian wanting to build a virus or bacteria that kills everyone. There's also religious cultists. It's worth reading about the Aum Shinrikyo cult. They wanted to start a global pandemic, and they believed for religious reasons they would be spared.
As far as the possibility of being able to design a bioweapon that only targets people of a certain ethnic/ancestrial background, my impression is that is very difficult, but I wouldn't dismiss the possibility. Some experts seem very concerned about it.
Hi Dan, thanks! I'm not saying omnicidal agents don't and won't exist. Omnicidal ones would be most interested in the large investment bioweapons need, hence why not-being-omnicidal is a big filter.
What I get at more is that people underestimate how much work, expertise, and coordination is needed to pull off a bioweapon that can kill a considerable number of people. I use Aum Shinrikyo in later essays as an exampl: even with a billion dollars and a collection of experts, they couldn't pull off even a simpler bioweapon successfully.
Part 3 will explain more of what an omnicidal group would have to pull off.
Glad you're looking at this, Abi. This always seemed to be a strange fear for people to be constantly mentioning. It presupposes that the pre-AI bottleneck for mass bioweapon deployment was the molecular design, which just never passed the smell test.
Thanks Vince!
Hi! Your framing around cost-benefit logic in weapon selection was the part I found myself pushing back on most, and I wrote up some counterarguments here: https://thecounterfactual.substack.com/p/contra-abi-olvera-on-ai-biosecurity
Would be curious for your takes / response!
Thanks for this extremely thoughtful response! I'll jump in with a quick reply because Abi is mostly offline for the next few days, and then I'll let Abi respond in detail.
The key idea that emerged from Abi's research is that there are bottlenecks which do *not* seem likely to be significantly eased, at least in the near term, by current developments. (You cite some of the most important, such as cloud labs and DNA synthesis providers.) Abi based this work on discussions with multiple people from the AI / biosecurity community (some of whom are named in the acknowledgement at the end of the post), including feedback on the final draft before publication. The next couple of installments will go into more detail regarding those bottlenecks, and we'd love further feedback.
You also mention the emergence of contract research organizations (labs-for-hire). This isn't a topic I know much about, and I am not certain whether it came up in discussions, so I'll have to wait for Abi here. I'll also defer to Abi to comment on precisely how her findings relate to the Virology Capabilities Test results; I know this was on her radar, and again the upcoming installments will shed more light here.
Thanks again!
Thanks Steve! Really appreciate the engagement, and I want to reemphasize that Abi's project is genuinely a valuable contribution; the practitioner perspective has been missing from this conversation, and I'm glad she's been doing interviews and discussions with people in the field!
I'm interested to see the next installments get concrete on specific bottlenecks that remain, but I think my concern might be less about any single development and more about compound effects of being able to route around *any* of the bottlenecks that used to serve as barriers to bioweapons development.
Also glad CROs are on the radar. Looking forward to the rest of the series!
[note: cross-posted]
Thanks Sophie for this follow up!
Your point 1 rests on assumptions that part 3 addresses at length. If you send me your email via SubstackDMs (or however you're comfortable). I'd be happy to send for your thoughts and feedback, as well as the other parts! :)
I'd love to know more about which viruses you envision in the airport example. A more specific example would be helpful because, depending on what type of virus you're proposing, the core bottleneck is different.
An existing virus faces the same frictions it always has: symptomatic spread gets detected fast, and killing people quickly limits transmission. It'd be very expensive way to kill a dozens or a hundred as you note.
An engineered novel virus raises a different problem: we don't actually know whether engineering a pathogen to behave exactly as intended is possible outside of a massive, multi-institutional team doing years of physical testing on humans. To career biologists, it sounds like "anyone can build nuclear weapons because the blueprints are online." It's technically true in a narrow sense, but not really in the physical world.
The biology field has an entirely different, maybe more difficult, nature than building a nuclear weapon or a bioweapon: you don't know whether the end goal you're aiming for (the theoretical virus) is viable, and if it is, whether it works as you expect with any part of the immune system or human biology. These three different problems make the type of physical experiments you need very very large, potentially infinite. The problem is hard because you don't know whether the endpoint is achievable, so you have no way to know if you're really much closer to a novel pathogen.
Aum Shinrikyo had a billion dollars and multidisciplinary experts and still couldn't produce a bioweapon. They had to turn to a chemical weapon.
We do actually approach cybersecurity this way. Lots of things are legal because "common actors" don't use it. Salad bars are a useful vector for bioattacks (see Oregon e.coli attacks) but they're still legal because it's so rare for people to want to do it. Even in cybersecurity, practitioners note that there are a ton of easy-to-exploit vulnerabilities, but people don't exploit them because 1. money laundering is annoying and human-work-intensive, and 2. people with hacking skills tend toward more lucrative cyber-related work, like working with state actors, for example. See Meta's former cyber person Joshua Saxe explain the actor model in cyber: https://joshuasaxe181906.substack.com/p/exploits-dont-cause-cyberattacks.
For context, I'm explaining why much of the biosecurity world is focused on institutional actors, not "groups of laypeople" or "lone terrorists". My post aims to clarify *why* biosecurity folks are focused on the most capable attackers. A lot of headlines give the impression that "any day now, any person can be a bioterrorist".
On the new technologies section:
Cloud labs — can you be more specific about the concern? If dealing with a known virus, how would they hide from the cloud lab that they're working with a known virus? A cloud lab does the highly personalized work of figuring out "how to boot up a virus" process without asking why someone is making a virus? If it's not a known virus, getting from DNA assembly to a functional pathogen requires potentially infinite iterations if you're dealing with anything novel. When dealing with a novel pathogen, how would they know which viruses are compatible with life (not human life)? If you cross that very high obstacle, you'd need many willing humans to figure out which viruses (if any are viable with biology) are viable with human biologies and immune systems.
DNA synthesis — this touches only one part of the bioweapon creation process, and not the hardest part. It gets a lot of attention because it's where AI intersects, but that's not the same as it being the key barrier.
CROs — agreed regulatory oversight matters here (and in cloud labs), and I don't think anyone is arguing against it.
On the 94% virology expert survey: I was glad to see the focus on lab work, but the questions may not capture what actually makes virologists dangerous or not. Multiple biologists told me virology is more like surgery than other STEM fields — the expertise is organism-specific, batch-specific, built up over decades of slow, hands-on learning. The survey gets at a sliver of laboratory protocols that are written down and some tacit knowledge. But the tacit knowledge required for each specific organism is huge and it's unclear whether the test gets at even 5% of it (random number, low confidence, could be <1% could be 15%). The tacit knowledge is the reason why the only person who succeeded at anthrax was a world expert on anthrax.
To be clear, I agree with your advocacy for more biosecurity efforts. But I believe your conclusion has unintended consequences. You said "arguments that downplay AI biosecurity risk— even unintentionally— risk contributing to policy inaction on a problem where the cost of delay could be extremely high."
My response on that: If the AI biosecurity risk is lower, but advocates overstate the risk, then it'll be harder to know when advocates are not crying wolf. It's also not good for group epistemics to see counterarguments as risky. It's a disservice to the field of AI-biosecurity. To get stronger arguments and be someone who governments reach out to, it's pretty core to understand why practitioners downplay it. It's the same reason the government will downplay it. Keeping "downplaying" types of evidence out of public discourse just harms the epistemics of people outside of government and makes it less likely that advocates will affect people inside of government.
Can't wait for your thoughts on the next parts! Do send me your email so I can send the Google Docs. :)
[cross posted]
This thread is super interesting! Did you continue the conversation in DMs/emails? Would be interested to read it if so (if you're happy to share)
Looks like the same topic (broadly) as this, if anybody is interested in a more biology-centric view: https://www.owlposting.com/p/reasons-to-be-pessimistic-and-optimistic
(no connection to the author, just a reader)
We've been thinking about this a lot at Radvac, the biosecurity nonprofit where I work. I personally think that AI-enhanced biorisk is the largest existential risk we currently face, and will likely be the dominant risk over the next decade. I also think it is very neglected, especially compared to AI safety, which has had an influx of very large numbers of highly skilled people in recent years.
I am mainly worried about actors that do want to kill everyone, including themselves. Emil Torres called these "omnicidal agents". I noticed your piece doesn't address this risk at all. Omnicidal agents are not a theoretical possibility, we already have examples of environmentalists who think that humans are a "cancer" on the planet and are in favor of human extinction. There are also negative utilitarians who are pro-extinction. It's very easy to imagine a mentally-destabilized environmental extremist or negative utilitarian wanting to build a virus or bacteria that kills everyone. There's also religious cultists. It's worth reading about the Aum Shinrikyo cult. They wanted to start a global pandemic, and they believed for religious reasons they would be spared.
As far as the possibility of being able to design a bioweapon that only targets people of a certain ethnic/ancestrial background, my impression is that is very difficult, but I wouldn't dismiss the possibility. Some experts seem very concerned about it.
For more on omnicidal agents, see https://www.sciencedirect.com/science/article/abs/pii/S1359178917302859 and let me know if you'd like me to send you the PDF.
Hi Dan, thanks! I'm not saying omnicidal agents don't and won't exist. Omnicidal ones would be most interested in the large investment bioweapons need, hence why not-being-omnicidal is a big filter.
David Manheim notes however that even Aum Shinrikyo wasn't really omnicidal, since they saw themselves as people who should and would survive. Their attack in the Subway was also mostly directed at police: https://thecounterfactual.substack.com/p/contra-abi-olvera-on-ai-biosecurity?utm_campaign=comment-list-share-cta&utm_medium=web&comments=true&commentId=241171570
What I get at more is that people underestimate how much work, expertise, and coordination is needed to pull off a bioweapon that can kill a considerable number of people. I use Aum Shinrikyo in later essays as an exampl: even with a billion dollars and a collection of experts, they couldn't pull off even a simpler bioweapon successfully.
Part 3 will explain more of what an omnicidal group would have to pull off.
OK, thanks.. I'll look forward to reading your findings with great interest!