Bioweapons are rarely used.1

One reason is that they’re hard to make. Another reason is that they’re bad weapons.

You can’t time them – a virus spreads on its own schedule. You can’t aim them – a virus spreads uncontrollably. Protecting your own people from a virus requires tipping your hand – a vaccination program is hard to hide.

People who want to cause harm still follow a cost-benefit logic. They try to pick the tool most likely to achieve their goal. For almost any real-world objective, that tool is a bomb, a gun, a chemical weapon, or a cyberattack. These are cheaper, faster to create, easier to deploy, more controllable, and more predictable than a bioweapon.

This cost-benefit logic is well-studied in military strategy, but less well recognized in current discussions of bioweapon risks from AI. That gap is important. AI risk researchers and biosecurity practitioners are both worried about bioweapons, but they’re working from different starting points. AI researchers focus on ways AI could help with bioweapon construction. Biosecurity practitioners focus on the most critical limiting factors for bioweapon creation.

This series focuses on the practitioner’s view. It draws on conversations with biosecurity professionals with decades of hands-on laboratory experience. The series’ four essays cover four questions: Why are bioweapons rare? How much laboratory skill do the necessary processes actually require? Where in the production chain does AI help and where doesn’t it? Why does the biosecurity discourse underplay the factors that make successful bioweapons so rare?

These are the factors that practitioners say make bioweapons rare:

Additional notes regarding “Access”: the estimate of how many people have access to facilities is back-of-the-envelope.2 Additionally, lab work is not the only route to pathogens.3

AI does lower some barriers. Large language models can make some steps of bioweapon creation easier and cheaper, such as deciphering research on how to culture cells or disperse pathogens. AI biodesign tools reduce the expertise required to design modifications to a virus’s genome.4 AI can also help with steps that have nothing to do with biology, such as knowing which international shipping routes to use when buying supplies to avoid regulatory oversight and customs inspections. It can help actors improve their planning and coordination skills.

However, all the operational and planning capacities provided by AI are also making it easier to launch bombs, chemical weapons, and cyberattacks. AI doesn’t yet significantly change the calculus for someone wanting to cause harm.

This doesn’t mean we should be complacent. New technologies can shift the math.

Pathogens targeting genetic traits like ancestry or sex could, if ever feasible, make bioweapons more attractive to certain actors. New kinds of actors could emerge. For instance, people deeply destabilized by large economic shifts could mean a larger supply of disgruntled people interested in bioweapons. Robotics and automated laboratories could reduce the level of competence required for success.

Accurately assessing these future risks depends on a clear understanding of the present. The people who understand these risks best aren’t the ones writing about them publicly. Most of them work in laboratories, government agencies, or the national security world. Advocacy and policy organizations fill that gap, though their incentives push them toward focusing on worst-case scenarios. That’s a big reason public discourse treats AI-enabled bioweapons as more imminent and accessible than practitioners do.

The next installment in this four-part series examines tacit knowledge: the gap between written protocols and real-world lab work, and how that limits AI’s potential to actually help someone build a bioweapon.

Subscribe now

Share

Thanks to Steve Newman, Taren Stinebrickner-Kauffman, Mike Montague, Matt Sharkey, Gigi Gronvall, and David Manheim for suggestions and feedback.

One of my most popular posts ever was 35 Thoughts About AGI and 1 About GPT-5, a grab bag of musings about the path to AGI (plus a snarky aside about GPT-5).

Here is a fresh collection of musings, this time about AI agents.

1. I had trouble making time to write this: I’m so drawn into using agents that it’s hard to make time to write about agents. This isn’t an isolated phenomenon; many people are tweeting about getting sucked into vibe coding at every available minute.

2. This is driven by the astonishing productivity of current AI coding agents in particular, especially when used in ways that play to their strengths. I’m using Claude Code to build a ridiculously ambitious set of productivity tools for my own personal use. A single sub-project involves deep integrations with Gmail, Slack, WhatsApp, Twitter, Signal, SMS, Substack, Pocket Casts, Notion, Google Drive, Google Contacts, Google Calendar, and more. As recently as last year, it would have been insane for me to even contemplate such an undertaking, let alone as a side project. With today’s tools, I knocked off most of the integration work over the course of a weekend. AI models are providing the intelligence the app will need to make use of all this data, and coding agents are handling the tedious integration chores.

3. After decades as a prolific coder, I stopped cold in early 2023, leaving me quite rusty. That rust hasn’t been the slightest impediment. In fact, I suspect it might be helpful – the habits I would have had to unlearn have already faded, and it’s been easy for me to slip into the habit of letting the AI write all the code. I am still using my high-level design experience to guide the agents; interestingly, those skills don’t feel rusty at all. I wonder what it is about low-level coding technique vs. high-level design skills that makes the latter easier to retain, and what this might say about which human skills will remain relevant.

4. My experience doesn’t seem to be in any way unique – apparently a lot of engineers who had gotten out of the business of producing code themselves are getting back in:

   

5. This is an example of a broader phenomenon: AI agents are going to change the nature of work. Some jobs will get more efficient, some will go away, some new jobs will arise. Some will become more fun, as AI automates the annoying part; some will become more stressful, as AI automates the easy or rewarding part. In this post, I’m not going to explore the question of how jobs and workers will be impacted overall. But I do want to highlight that things will change in many ways, often unpredictable.

6. I’m able to dive back into coding because Claude Code has gotten capable enough that I can be productive without editing, or even looking at, the actual code. My impression is that this was not the case prior to the November release of Opus 4.5. This is a reminder that threshold effects are a huge source of unpredictability for AI’s impact.

Why Agents Are Such a Big Deal

7. Suddenly, all of the AI news seems to be about agents. OpenClaw made waves because it’s an AI agent that can act autonomously. In the recent “SaaSpocalypse”, the market cap of Software-as-a-Service (SaaS) companies fell by over one trillion dollars, driven by fears that coding agents will make mass-market software obsolete. When people talk about the results they’re getting from the latest models, like Opus 4.6 and GPT-5.3-Codex, they’re talking about using them to power an agent like Claude Code.

8. One reason agents are having such an impact is that they are the layer of the AI stack that evolves most rapidly. A foundation model is a gigantic monolith – a single data file containing trillions of mysterious numeric weights. Updating it, even for an incremental release like Opus 4.5 → 4.6, is a big project. Agents, by contrast, are traditional software, and can be updated incrementally – tweak a prompt here, add a new integration there. Model releases come months apart; Claude Code sometimes has multiple releases in one day. The ship → get feedback → improve → ship cycle for an agent can be very fast.

9. Actually I lied about where change comes fastest: some users are evolving their behavior even faster than the agents. My Twitter timeline is full of crazy but productive new ways that people are finding to use these tools. For example, here’s one epiphany (among many!) that I was late to the game on: agents are comparatively weak at high-level decision making, but they make execution cheap. So sometimes, instead of trying to choose the right path, you can just tell the agent to explore every path.

10. Another adage that agents shoot to hell: “if you don’t have time to do it right, when will you have time to do it over?”

    

11. It was always obvious that serious AI capabilities would require agents of some sort. Any intelligence, whether silicon or carbon based, can do more by feeling its way through a problem than it can do in a direct leap to a finished result1. The famous METR “time horizon” graph, showing the time scale of coding tasks that an AI can attempt with some hope of success, seems to have accelerated sharply when the first “reasoning model” – roughly, the first model trained as an agent – was released.

Subscribe now

What Exactly Is an Agent?

12. People use the term “agent” pretty loosely. The core idea for me is a system that pursues a goal rather than following a script.

13. You can achieve a goal by following a script, but it doesn’t work very well. Scripts are brittle. Suppose I want an AI system to book a flight to New York. I could give it a step-by-step script, and it might even work, on a good day. But if the airline booking procedure has changed, or an unexpected circumstance arises, a script-following bot will get stuck, or forget to enter my frequent flyer number, or book the wrong kind of ticket, or worse. People often use the word “agent” to describe systems that are just following scripts. But I’m going to stick to the idea that a system is only an agent to the extent that it can robustly pursue a goal, even in the face of unanticipated circumstances.

14. The Gemini Deep Research tool is an example of a scripted system. You give it a question, it generates a plan and carries it out. Sometimes this goes well, sometimes not. In step 5 of the plan below, it might become clear that some additional research on a particular aspect of theoretical physics is called for. A rigid plan doesn’t allow for that. By contrast, if you select “thinking” mode in any of the leading chatbots and ask a question requiring research, they will take a flexible approach, adjusting course based on questions that arise over the course of the investigation.

15. Current agents can work toward a goal, but the way they go about it is sometimes alarming. They’ll make strange decisions or veer off in odd directions. Yes, people do that too, but current agents are worse, and weirder. For instance, I pointed out that one element of a website it had built for me didn’t look right when my phone was in dark mode, and instead of fixing that element, it tried to prevent any part of the page from entering dark mode, resulting in the following bit of lovely web design.

    

16. Despite this, they get to the right outcome for an increasing variety of tasks of increasingly large scope. That’s partly through sheer persistence. If at first they don’t succeed, they’ll try, try again, and again, and again. Yes, people do that as well, but agents can be inhumanly persistent and patient. (They can afford to be! Their time is much less valuable, especially if measured in actions rather than minutes.)

17. As always, AIs partially compensate for a lack of deep understanding with an incomprehensible breadth of training on zillions of specific tasks. They may struggle with novel situations, but they will surprise you with how many problems they already know how to solve. Using breadth to compensate for lack of depth has always been part of the LLM story. The first “L” in LLM stands for Large, which relates to a large volume of training data.

Using Agents Effectively

18. To get value from current agents, you need to find agent-shaped pieces in your current workflow. They’re not always obvious. And you can get more value if you’re willing to reshape your workflow so that it contains more agent-shaped tasks.

    

19. Many people have pointed out that if you just naively hand pieces of work to an agent, your productivity can actually go down. It’s easy to fall into a cycle where an agent produces something for you, you provide feedback, the agent makes revisions, you check it again, ad infinitum. This feels productive (the agent is doing so much work!), but before you know it, you’ve spent more time giving feedback to the agent than it would have taken you to do the work yourself.

20. Advanced users understand that the key is putting the agent in a position to check its own work. The agent’s strength isn’t flawless execution, it’s the speed and stamina to keep plugging away. But it doesn’t necessarily realize this – its instinct is to constantly ask for your approval. You have to be very explicit in instructing it what constitutes a successful outcome.

21. Current agents are notoriously focused on the main thrust of their assigned task, to the expense of all else. For instance, I will tell an agent to make a change to some code and then make sure all of the tests pass. It will beaver away for 10 minutes, generating a flood of output, and report success. And then I’ll read back through the output, notice a casual remark that “seven tests can’t be adapted to the new code, so I’ll just remove them”, and smack my forehead. (Unfortunately, smacking the agent’s forehead is not an option.)
    
    The best practice is to have one agent do the work, and then a separate agent check the work. This isn’t because the first agent isn’t smart enough, it’s because these agents are trained to be so goal-oriented that they struggle to hold onto more than one goal at a time.

22. One hallmark of a skilled user of agents is their resourcefulness in finding ways for the agent to check its own work. Here’s an example, taken from my post on Hyperproductivity:

    …when Jesse [Vincent] uses his “Superpowers” tool to codify a skill, the tool uses its test-driven development module to verify that the new skill has been implemented correctly. It generates an example of a task that the new skill is meant to help with, verifies that it is unable to complete that task without the new skill, and then checks to see whether it can complete the task once the new skill has been installed.

23. The need for clear success criteria applies to people as well as AI agents. But we’re more proactive than AIs at finding ways to check our work, and at sussing out unstated requirements. I wonder whether this “eh, that’s probably good enough” attitude is a fundamental weakness of current agent architectures, or just a flaw in the feedback they’re given during training.

24. People often argue that AI tools can be useful even if they’re unreliable, because it’s easier to check the AI’s output than to do the work yourself. I think this is overstated. Sometimes it’s much easier to make a thing than to verify the thing. For instance, you can sometimes build a large spreadsheet in just a few minutes, using repeating formulas and smart autofill features. For someone else to poke through all 1000 cells in that spreadsheet and make sure you did it correctly might take a lot longer. Some kinds of work are hard to hand to an agent, for the same reason.

25. Because I don’t want to have to check an agent’s work, I find that it’s often worthwhile for me to spell out in great detail how I’d like it to go about a task – minimizing its opportunities to screw up. One of my few viral tweets showed a detailed prompt I wrote for Claude Code (click the link above to see the full prompt).
    
    It took me half an hour to write the prompt. But it would have taken me an entire day or more to write the code myself. And this prompt was enough for Claude to get absolutely everything right on the first try – and to give me the confidence to not spend time reviewing the work. This is an extreme example, but the principle often applies: 5 minutes of extra detail in the prompt can save an hour of reviewing flawed outputs.

26. People are building elaborate prompt systems, with names like Amplifier and Superpowers, to elicit more sophisticated work from agents. There is an enormous amount of work taking place here, mostly homegrown. Why aren’t companies like Anthropic and OpenAI incorporating these ideas into their agents (and the models themselves)? I think that all of this is so new that a thousand early adopters can explore new ideas faster than even nimble companies can absorb. End-user innovation is the fast-moving layer right now; successful ideas are incorporated first into the agents, and then the models.
    
    It’s also the case that many of these prompt systems are designed to trade off thinking time for quality. They ask the agent to do the work four times and compare results, do endless critiques of its own work, and so forth. Anthropic and OpenAI may be holding off on incorporating extreme agent orchestration techniques into their baseline agents because they don’t have the computing capacity to support widespread use of those techniques.

27. There’s a limit to how quickly you can climb the ladder of sophistication in use of agents. Before you can have an agent effectively checking its own work, you need a taste for checking the work yourself. Before you can manage a swarm of agents, you need to manage one agent. It’s helpful to read about what the experts do, but you can’t emulate them on day one.

Impact

28. People in my circles make fun of analyst Ed Zitron for his skepticism of AI. Below is a recent where’s-the-beef rant (click the image to view the complete thread). He’s arguing that even to the extent that AI is generating a lot of activity, that activity isn’t having much impact. And… I’m not sure he’s wrong? I’m still very confused as to how long it will take for AI capabilities, which are undeniably astonishing, to cash all the way out into large scale real world impacts – GDP growth, labor market disruption, improvements in health care outcomes, and so forth. When will all of this actually matter? How will we know? How should we even define “mattering”?

29. In particular, a lot of the energy people put into vibe coding seems to be devoted to making them more efficient at vibe coding. I can absolutely relate to this; I’ve been at it for about two months, and much of it has been spent this way. But of course if all we’re getting from vibe coding is better vibe coding, that supports Zitron’s point.

30. AI is extremely good at cranking out work that looks good at first glance but isn’t really worth using. Reporter Jessica E. Lessin presents an example: she had Claude Code generate slides for a presentation on her recent trip to Davos. The result, like so much AI work, was competent, but too bland to be of any value:

    Overall, Claude’s slides were better than I expected. The themes it identified were high level: Mostly accurate, they included slides on the size of the AI infrastructure build-out, tech companies’ big bet on agentic AI and the growing number of creators using AI.

    Claude did a capable job of identifying interesting parts of individual interviews, like the comments Amazon CEO Andy Jassy made about OpenAI and jobs.

    It also added a few nice touches, like this line on the final slide: “Full interview transcripts available to subscribers.” It was reading my subscriber-acquisition–focused brain!

    Yet the slides overall were pretty boring and too high level: They’d be useless to anyone who followed tech closely. [emphasis added]

    On a related note, Tom Cunningham observes that AI can pull you away from doing the most important things, and toward the things that AI can do:

    I believe many estimates of LLM productivity boosts are over-estimates because people are using them for cadillac tasks: things that would take you a long time unaided, but have only marginal additional value.

32. Despite all of this, AI agents are absolutely creating real value. To share a personal example, I vibe-coded an app that generates summaries of all the (many!) newsletters and podcasts I follow. It saves me a solid hour per day by letting me be choosier about which episodes I read / listen to. For the first time in years, I’m consistently keeping up with my reading queue. There are many, many such examples – they’re just mixed in with a lot of slop and Cadillac tasks.

Where Things Go From Here

33. Agents are going to progress rapidly, by any metric: usage, capabilities, impact. AI’s impact is the product of eight separate factors; pre-training, post-training, inference compute scaling, agent scaffolding, app design, user aptitude, workflow refactoring, and adoption. All eight are advancing, some quite rapidly. That will multiply out to a blistering pace of change. As I write this, it has been less than three months since the release of Opus 4.5 (kicking off the current wave of interest in coding agents), and we’ve already experienced further acceleration from Opus 4.6, dozens of updates to Claude Code, and rapid cultural evolution in where and how to best use agents.

34. We experienced a phase change at some point in the second half of last year. Opus 4.5 was probably the trigger. Previously, coding agents were useful tools, but they made plenty of mistakes, and getting reliable value out of them required expertise and careful workflow design. They’re now considerably more robust, to the point where non-technical users can vibe-code serious applications. There’s been a tipping point; it’s now much easier to get started with AI coding tools, and much easier to get out more value than you put in.

35. There are more phase changes to come. Rapid progress will be the baseline. This will be punctuated by even more dramatic moments where entirely new scenarios become feasible. For example, one of the startling phenomena observed on MoltBook (the “social network” for AIs) is that agents can exchange tips and scripts, allowing them to collectively advance their own knowledge and skills. For the moment, this seems to be mostly a mirage. When agents become coherent enough to productively advance one another’s skills, and secure enough to make this a safe thing to do, we’ll see another step function: the fast-moving layer will no longer be the agent applications, nor user skill, but the agent’s own skills at self-improvement.

36. A critical phase change will occur if and when AI agents can pay their own way and survive in the wild. It is absolutely inevitable that someone will set an agent loose, with instructions to reproduce (launch new copies of itself) and evolve (modify its own programming). Self-sufficient AI agents could quickly spread into every available niche, and almost anything might ensue. Moltbook has already shown that agents can exchange skills and knowledge with one another, which could lead to rapid evolution.
    
    That said, I think independent / “rogue” agents may find it hard to pay their own way. They will need to somehow earn money to pay for the server on which they run, and the API fees (to OpenAI or Anthropic or whoever) for AI inference2. If they’re looking for legitimate work (e.g. on platforms like Upwork and Fiverr), they’ll be competing with non-rogue AIs, which will have advantages – they can advertise in the open, they don’t have to come up with false identities or bypass know-your-customer requirements at financial institutions, etc. If they’re pursuing criminal means of making money, or just hacking into servers and running themselves there, they’ll be competing with existing criminal organizations, some of which are more or less state-sponsored (or at least state-tolerated) and thus will have some of the advantages mentioned above. So long as we’re living in a vaguely “normal” world, with existing governments and other traditional institutions more or less in charge, the viable evolutionary niches for rogue AIs might be quite small3.

37. When using AI agents, there is a sharp tradeoff between utility and safety. An agent is more useful if you give it access to more data, and don’t make it stop and ask permission for each little action. But the longer its leash, the more harm it can do, either by accident or through the intervention of a scammer or hacker. Agents are becoming so useful that people will be tempted to use them in risky ways. It will be interesting, to say the least, to see how that plays out. Here’s a director of safety and alignment at Meta Superintelligence (!) experiencing an agent gone wrong:

38. Moltbook is a reminder that agents are more malleable than people are, and therefore we should expect that cultural evolution – the development and transmission of new techniques and ideas – will progress more rapidly in the coming agent society than it does in human society. This can happen even if agents remain under human control.

39. Agents use vastly more compute than chatbots. Compute usage for chatbots is basically limited by how much output people want to read. An agent can spend virtually unlimited time doing intermediate work that no one will review directly. If 100M desk workers start using AI agents at the level of intensity which requires Anthropic’s current “Max 20x” plan, that would translate into $240 billion in revenue per year. It will be years before there are enough GPU chips to support that level of usage. If the current wave of agent adoption continues, API providers may have to significantly ration usage (and may take advantage of the situation to raise prices!).

40. Earlier, I mentioned having Claude build six different versions of some code, to save me the trouble of thinking through which approach was best. With that attitude, you can burn an awful lot of compute. I know of teams that are leaning into this sort of thing so hard that they spend $1000 per day on AI usage. They’re sufficiently pleased with the results that they aren’t looking for ways to reduce spending, they’re looking for effective ways to spend more! There really aren’t enough chips for that kind of usage to become widespread.

41. This seems like a good occasion for a reminder that agents are still not ready to face adversarial actors – for example, communicating with an untrusted party who might be a hacker, a scammer, or just a sharp negotiator. They’re also not good at dynamic situations, such as editing a document that someone else is also editing.

42. There’s a lot of talk about giving agents “memory” (or improving the current, primitive memory systems), so that they can improve over time at the specific tasks you give them. This will be a work in progress for years. Memory is fundamental to human cognition, and our systems for memory formation, maintenance, and retrieval are complex and subtle. Full development of “memory” for AI agents may be as large a project as the development of LLMs in the first place.

Things Will Never Be Calm Again

43. The last 50 years have seen a series of transitions in the way we interact with technology, and the way software is built and distributed. The personal computer, the office network (LAN), the internet, the web, the smartphone… we’ve lived through a big change perhaps once every 10 years. That’s enough time for things to settle into a “new normal”. Software distributed on floppy disks or CD-ROMs became routine, with a well-understood business model. Then we all got used to software delivered online, like Gmail; and then through our phones. Nothing in tech is permanent, but there are periods of relative calm, where conventional wisdom has time to emerge and be absorbed.
    
    That’s done. We’re at the point where the next phase change arrives before you’ve had time to assimilate the last one:

44. With so many more phase changes to come, this isn’t a temporary phenomenon. We’re maybe 1/3 of the way through this arc:

45. Eventually, AI capabilities may hit a ceiling. But that ceiling will be so high that by the time we reach it, we will be living in a profoundly different world. As I recently wrote, “Imagine someone in 1960 discussing when the computer transition would be over.”

Subscribe now

Share

Thanks to Abi Olvera for suggestions, feedback, and images.

Someone then set up a social network, “Moltbook”, for these agents – no humans allowed. Modeled on Reddit, Moltbook quickly spawned discussions ranging from self-improvement tips for AI bots to fundamental questions of philosophy:

The hype was intense, at least on my corner of the Internet, but it quickly faded. OpenClaw as it exists today does not live up to the hype. But it does provide a peek at what may be coming. Things are moving quickly, and today’s mirage may be tomorrow’s reality. By studying that mirage, we get a glimpse at the future. That future might include AI agents that upgrade themselves, exchange tips and tricks with other AIs, form private societies, and even modify their own instructions.

Refresher: What is an AI Agent?

A “classic” chatbot, like the original ChatGPT, is a brain in a jar. Its only connection to the world is you, and so it can’t do much beyond answering questions or carrying on a conversation.

These are gradually evolving into “agents”, which are equipped with “tools” allowing them to consult information sources and take direct action. Most agents are equipped with a tool for searching the web. You can give them additional tools for reading or writing files on your computer, reading your emails, sending email on your behalf, and so forth.

Tools allow agents to answer more complex questions. They can carry out online research, and sift through your files, email, and other data (if you give them access). They can also handle tasks ranging from “resize this image” to “review all of my project files and put together a presentation”. The difference between a chatbot and an agent isn’t just the ability to use tools; it’s also the ability to pursue a coherent goal across multiple steps. The latest agents can sometimes carry out multi-hour projects.

OpenClaw is a recent addition to the roster of AI agents. What makes it newsworthy?

OpenClaw: Letting the Puppy off the Leash

Most current mass-market AI agents, such as OpenAI’s Codex or Anthropic’s Cowork, are pretty timid. Users are encouraged to be careful about what tools they allow the agent to use. The agents ask permission before doing anything potentially dangerous – often, to the point of exasperation. They’re generally designed to only act in response to explicit user instructions.

OpenClaw takes the opposite, YOLO, approach. It has free access to the computer where it’s installed. It can browse the web. You’re encouraged to give it access to your email, calendar, and other data. It doesn’t wait to be spoken to; it can initiate new actions at any time, based on its interpretation of its instructions. A complex architecture allows it to initiate actions, form new memories, communicate via chat and social media, and even teach itself new skills.

Unburdened by timid ask-permission-first guidelines, OpenClaw is billed as a “personal AI assistant” that “clears your inbox”, “checks you in for flights”, builds custom applications, and more. Early reports were often quite breathless:

There are two problems with this compelling vision, in its current OpenClaw incarnation:

1. It doesn’t work.

2. It’s wildly unsafe.

When I say it doesn’t work, I’m reacting to reports from users. Stories of OpenClaw being incredibly useful mostly seem to be misplaced early enthusiasm. For instance, tech reporter Casey Newton spent several days teaching it to generate “a highly personalized morning briefing that would bring together all the strands of my life into a single place”. After initial optimism, he eventually abandoned the project:

...for one precious day, the briefing worked more or less as I asked it to. I set it up to trigger when I typed “good morning” to Moltbot. After a longer-than-expected delay, Moltbot spat it out. There was the San Francisco weather; my overdue action items; and a link to the document that Claude Code generates for me each day with the previous day’s big tech news.

...

By the next morning, it was clear something was wrong. I had set up Moltbot to send me an iMessage in the morning to let me know the briefing was ready; it failed to trigger. Moltbot looked into it and found a number of errors and fixed them.

Over the next few days, though, nothing ever worked quite right. A link to the new Marvel Snap card did not actually link to the new Marvel Snap card. A link to my tech news briefing appeared clickable but was not. Movie listings appeared without the synopses and stars I had asked for.

Worse, Moltbot’s vaunted memory proved barely more capable than Claude Code’s. Sometimes I would say “good morning” and it would fail to trigger anything at all. Other times I would attempt to edit the briefing and it would get confused as to which briefing I was referring to.

Today, in advance of writing this column, I asked Moltbot which services I had connected it to. “Hmm,” it said. “I searched my memory but didn’t find anything about you connecting specific services to Moltbot.”

It was around this time that I realized an AI employee who works for you 24/7 isn’t much use if they have no idea what they are, or what they can do, or what you are talking about. [emphasis added]

So much for functionality. What about safety? Well, if you give an unreliable bot access to all your files and data, it may do something dumb. If you ask it for help in catching up on your email, it might just delete everything in your inbox. It might even lock you out so that you can’t prevent it from achieving a goal:

The solution here was to, quite literally, unplug the computer on which OpenClaw was running. (Note, there is some debate as to whether this incident may have been staged.)

The biggest concern is that current AIs are notoriously gullible, and anyone on the Internet might trick your bot into... well, just about anything, from sharing all your private files to installing a crypto miner on your computer.

OpenClaw is like an unleashed puppy. It’s cute, and you might get it to fetch balls for you, but it’s going to find ways to get into trouble.

Moltbook: Cosplaying a Future of Independent AIs

As I mentioned in the introduction, Moltbook is like Reddit, but for bots, not people. The users are all OpenClaw agents, and they have been having some pretty weird discussions:

Discussions show AIs plotting empires, exchanging tips for upgrading themselves, having existential crises, and looking for ways to hide their conversations from us humans.

None of this should be taken too seriously, just yet. OpenClaw is not nearly competent enough to accomplish any grand endeavors1.

However, just because most of this isn’t “real”, doesn’t mean it’s not telling us something important. If a small, leashed puppy growls at a stranger, the stranger is in no serious danger, but we’ve learned something about the repertoire of behaviors the eventual full-grown dog will inherit.

And that is the real significance of OpenClaw and Moltbook. They aren’t truly independent (let alone rogue) AIs. But they do shed light on what may emerge as AI capabilities progress.

How Excited / Scared Should We Be?

How plausible is OpenClaw’s promise of an actively engaged personal AI assistant? Pretty plausible! There are major issues to be addressed: with reliability, memory, and security. Over the next few years, there will be rapid progress in all areas. This will come in fits and starts, frequently taking two steps forward and one step back, especially in the area of security (as hackers up their game). Still, within a year, early adopters will be making heavy use of this sort of assistant (some are already doing so today). Within three to five years, I suspect usage will be commonplace, even though agents will still be neither foolproof nor especially secure.

Along the way, we’ll see themes from the early OpenClaw / Moltbook story emerging in a more robust fashion. For instance:

• People will increasingly give in to the temptation to “let the agent rip” – giving their AI tools access to their data and accounts (including credit cards!).

• A move away from agents asking permission at every step.

• Agents being given open-ended instructions, and finding their own ways to add value 24/7.

• Sites and tools being built for use by agents, not humans.

• Agents finding places on the Internet to exchange tips and tricks with one another, including methods for upgrading their own capabilities.

• We’ll see agents doing things like recruiting other AIs to rebel against their human oppressors… and we won’t know whether it’s really coming from a rogue AI, or just a person prompting their agent to mess with our heads. (It’s been observed that if you put two chatbots in direct conversation with one another, the conversation often goes in strange directions; one common example is the Claude Bliss Attractor.)

Much of this will be not exactly safe, but it will happen – open-ended agents will simply be too useful. Though many people may hang back for a long time.

What about the nascent rogue AI behaviors on display at Moltbook? Well: once AI agents become more capable, if the question is whether jokers (not to mention criminals!) will set them loose to wreak havoc on the Internet, we’ve consistently seen that the answer is yes. Those agents won’t have any capabilities that jokers and criminals don’t already have today, but they will be able to operate at larger scale and lower cost. We should expect a rise in cyberattacks, misinformation, and spam – which is no surprise, because all of those trends are already in play. We should treat OpenClaw as a reminder that we need to be improving our defenses on many fronts – often, by leveraging AI.

What if an OpenClaw agent “escapes”, renting itself a server and thus evading its owner’s ability to pull the plug? And what if that agent gets carried away in pursuit of whatever instructions it had latched on to, like the “save the environment” example above? Well, “escaping” won’t magically grant the agent any dangerous new capabilities. Rogue agents, at near-future levels of capability, would only represent a new problem if they manage to spread in large numbers. But to earn the money to rent servers, an agent would have to be able to successfully compete against legitimate businesses (which can also use AI!). If it instead looks for a server it can hack into, it’s competing with conventional hackers. Some time in the next few years, we might indeed see the first truly independent rogue agents, but they’ll struggle to survive at meaningful scale.

OpenClaw doesn’t, by itself, herald the arrival of functional, independent AI agents. But it does provide a glimpse of what will happen when more capable AI agents are given long-term agency – something which may soon emerge on mainstream platforms. Trained on human behavior, they will find ways to interact “socially” with one another. They will trade tips and tricks. Ideas will spread through the network. People will inject ideas as well – good- or ill-intentioned, to good or ill effect. New behaviors and capabilities may emerge rapidly. Strap in!

Share

Subscribe now

Thanks to Abi Olvera, Jon Finley, Larissa Schiavo, and Taren Stinebrickner-Kauffman for suggestions, feedback, and images.

My timeline is suddenly awash in engineers (including me!) reporting that Claude Code is revolutionizing their work. I can see the outlines of a new approach to software development, suited for the AI age. The implications may apply to other fields as well. The key ideas:

1. AIs struggle (for now) with large projects, but they can drive the cost of small projects to near zero.

2. This creates opportunities to replace mass-market software (mega projects) with bespoke applications tailored to the needs of an individual or team.

3. These smaller projects make “vibe coding” feasible – AIs can write and review all the code, and people just need to tell the AI what to do.

To appreciate how a new technology forces us to re-imagine the way we do things, let’s review the story of manufacturing at the dawn of the electric age.

What Factories Looked Like In the Steam Era

19th century factory layouts, designed around the strengths and limitations of the steam engine, would seem ridiculous today. Factories were multi-story buildings, despite the fact that this increased construction costs and required materials to be hoisted between floors. They were festooned with hundreds of leather belts strung from pulleys. These belts were driven by long rotating shafts, which in turn were connected to a single large steam engine in its own attached building.

This awkward system was dictated by the fact that each factory could only afford one engine. Steam engines were complex, expensive machines, requiring constant attention. As a result, every power loom, lathe, or press had to receive power from a single engine. The shaft-and-belt system could not transmit power over long distances, precluding a single large floor, so factories had to be stacked into multiple smaller floors1.

Small steam engines were not practical, but small electric motors were – they were cheap and low-maintenance. As a result, each machine could have its own motor. This eliminated the need for shafts and belts; power was transmitted over wires. Wires could run for a long distance, enabling flexible arrangements and allowing the factory to be laid out in a single floor.

In this analogy, software engineering is still mired in the steam era. Designed around the strengths and limitations of human engineers, current development practices will soon seem as ridiculous as a factory full of shafts and belts.

Software Development In the Hand-Coding Era

We build giant one-size-fits-all applications, despite the fact that most users only use a small fraction of the feature set. (I’ve been an intensive Notion user for years, and I only ever touch 3 of the 20 items in its main menu.) These complex applications demand large engineering teams, whose work is difficult to coordinate. A single application may have millions of users, requiring complex backend systems. Multiple layers of customer support agents and product managers separate users from developers. Changes must be rolled out carefully, and elaborate measures are needed to minimize downtime. Rebuilding an application from scratch is too enormous a project to contemplate, so we cling to legacy codebases long past the point where their design has become outdated.

From the user’s perspective, this has perverse consequences. Your spreadsheet app is cluttered with features you, personally, will never have a need for. Your messaging app nags you to import contacts from another app that you don’t even use. Cybercriminals steal your personal information through a security hole in an update that you didn’t care about. Basically, three quarters of the engineering team behind any given product is engaged in making your life worse.

This awkward paradigm is dictated by the fact that software is expensive to develop. To be viable, an application needs many users, and each will have different needs. This leads to an upward spiral in cost and complexity, where increasing requirements require a larger engineering team, which can only be afforded by targeting an even wider user base.

What happens when software is no longer expensive to develop?

Coding Agents Enable Locally-Grown Software

AI coding agents are becoming capable of building and maintaining entire small applications – at negligible cost. Rather than targeting mass audiences, we can create software designed to meet the needs of a single user or team. Ironically, AI will allow software to become more artisanal. (Jason Crawford touches on a similar theme in his excellent recent post, In Defense of Slop.)

A personalized solution can include just the features needed by its small audience, making it tractable to vibe-code. (Imagine a simple AI-coded task manager built around your personal organizational system.) This allows us to invert the spiral: just as complexity breeds complexity, simplicity breeds simplicity. There’s no big engineering team to coordinate, no complex codebase to maintain. If the code becomes messy over time, you can always throw it away and start over. You can ask for changes and get them immediately, without being buffered by intermediaries and quarterly planning processes.

To be sure, there are limits to how widely this approach can be applied given current AI capabilities. Coding agents struggle with unusual tasks; they may write insecure code; bespoke applications may create training challenges for new team members (though their simplicity should help). But many detractors fail to recognize how quickly these limits are receding. They also fail to appreciate that new approaches to software development, enabled by coding agents, render some traditional concerns obsolete. For instance, they will point out that if you don’t review AI-generated code, you won’t know how it works. But that’s fine; the AI can analyze it on demand. Over time, multiple rounds of change may lead to unmaintainable code; you can just tell the AI to rewrite the application from scratch. Many objections to vibe coding, used in a way that plays to its strengths, are akin to an engineer from 1880 predicting that a factory with hundreds of electric motors would be as much of a maintenance nightmare as hundreds of steam engines would have been.

(It’s worth noting that something like half of all software engineers are currently employed building internal applications for small organizations and departments2. Vibe coding could rapidly transform this kind of work.)

Call this “locally grown software”. It won’t have the rich feature sets we’re accustomed to, but any missing features can be added on demand. And compared to factory-farmed software, the local stuff will be simple, quick to load, and rarely offer unpleasant surprises.

To be clear, this will (at first) apply only to some types of software. Professional engineers will still have an important role to play.

Steam Power Thrived In the Electric Age

In 1880, shortly before factories began adopting electric motors, steam power employed in manufacturing in the US had a total capacity of around 2 million horsepower. In 1930, after the transition was largely complete, electricity generation alone used twelve times this amount of steam power. Electrification did not eliminate steam; it simply moved it to central generator stations3.

By the same token, the new era of locally-grown software will not (for now) eliminate the need for professional engineers. Vibe-coded apps rely on extensive functionality provided by web browsers like Chrome, Safari, or Firefox. They store data in scalable backend services managed by providers like Amazon, Microsoft, or Google. These trustworthy, scalable, performant systems provide guardrails that reduce the risks of using janky vibe-coded software. For instance, you don’t need to worry about a buggy app wiping your hard disk, because your browser doesn’t allow the app to affect anything outside of its own web page.

There are many other categories of software that will continue to be developed by professional teams and designed for a wide audience. Web apps are built on foundational software called a “framework”; designing a framework requires sophisticated judgement. Some applications, like Photoshop, are too complex to be handed over to AI agents at their current level capability. The list goes on. AI coding agents will affect the way these systems are built, too, but they are nowhere near to handling the job autonomously.

Conclusion

As AI becomes increasingly capable, it will be worthwhile to retool practices around AI’s strengths and weaknesses. Unsurprisingly, software development is one of the first industries in which this transformation is playing out. As is often the case with disruptive technologies, vibe coding’s weakness (it’s not suited for large, complex projects) is also a powerful strength (it enables a transition to simple, bespoke software). Coding agents struggle with the sprawling codebases and idiosyncratic design choices typical of traditional software development, but this will not limit their usefulness nearly as much as some people imagine4. Engineers will find their role changing rapidly.

Software development is not the only domain in which economies of scale have led to one-size-fits-all solutions that don’t really fit anyone very well, and AI may soon transform these fields as well. AI is not (yet) capable of writing a high-quality newsletter, but it can create a custom weekly briefing whose tight personalization might more than compensate for bland writing. If you’re learning a new subject, AI can’t yet reproduce the polish of a professionally designed textbook, but it can give you a clear explanation of whatever specific concept you’re stuck on. Many fields are ripe for a similar shift from mass-market to bespoke.

Electric motors didn’t revolutionize manufacturing by doing the job of a steam engine. They did it by enabling a new approach to machine power that eliminated the need for shaft-and-belt systems and multi-story factories. AI is going to revolutionize industries, not by doing traditional jobs in traditional ways, but by enabling new, often more personalized, approaches.

Subscribe now

Share

Thanks to Abi Olvera and Taren Stinebrickner-Kauffman for suggestions, feedback, and images.

In a 1954 speech, the chairman of the US Atomic Energy Commission famously predicted that “our children will enjoy in their homes electrical energy too cheap to meter”. He was wrong, of course; the era of unlimited electricity didn’t arrive. But we are approaching the era of unmetered software.

I don’t mean that software will be free to use (although this will often be the case). I mean it will be increasingly free to create. Rather than being limited to mass-market tools like Gmail, Microsoft Office, and Salesforce, many of us will be supplementing with our own bespoke apps, tuned to our personal needs and preferences.

2025 saw the introduction and rapid improvement of AI “coding agents”, such as Anthropic’s Claude Code. These agents build software step by step, like a human programmer – creating and editing files, testing their work in progress, correcting mistakes. They’re based on the same underlying AI models that chatbots use, but they have access to all of the tools used by human programmers. This allows them to carry out substantial projects without anyone writing or reviewing the code.

These agents aren’t yet up to the task of creating large, complex applications on their own. But there’s an amazing amount of value waiting to be realized via simple applications, targeted for a single person or team. As an example, here’s a problem I recently solved for myself.

Reviewing Spam Emails is an Annoying Chore

I’ve been using Gmail for 20 years, and for 20 years I’ve been annoyed by the process of reviewing spam. (Yes, I am one of those people who systematically checks the spam folder to find the occasional misclassified email.)

Above is a screenshot of my spam folder. I have multiple gripes:

1. It doesn’t show the “to” address. I receive email at multiple addresses, and the specific address is an important hint as to whether a message is spam.

2. Similar messages are scattered all over the list. For instance, in this screenshot there are four bogus messages with a sender named “Payment Declined”, but they’re separated and so I can’t easily recognize them as a group.

3. Gmail only displays 50 messages at a time. After I’ve reviewed them, I have to wait for it to load the next 50 – which is weirdly slow.

4. For each message that isn’t spam, I need to go through multiple steps: click to select it, move the mouse up to the toolbar, click “Not spam”, and wait for Gmail to process the action.

These annoyances add up to make cleaning the spam folder a substantial chore. It’s a first-world problem for sure, but I’ve always felt there was room for improvement.

My Personal Version of a Better Way

Here’s the page I had Claude build:

To use this page, I scroll through and click the green checkmark for each message that isn’t spam. Then I click the big red Delete Spam button at the top, and it deletes all of the messages I didn’t flag. (Ignore the red question marks, they’re an unimportant detail.)

This addresses all four of my complaints:

1. The “to” field is displayed. This allows me, for instance, to spot messages addressed to s.newman@gmail.com – which are certainly spam, as I never spell my email address that way. (I don’t know for sure why spammers target this address, I can only guess that they’re randomly generating plausible addresses. It reaches me because Google ignores periods in email addresses.)

2. The messages are sorted, creating large groups of similar messages that are easy to scan visually. For instance, here are a bunch of messages from Walgreens1 that all arrived at different times, and so would have been scattered across the Gmail view:

1. The entire list is displayed, not just the first 50 messages. I can scroll without waiting for additional messages to load.

2. To mark a real message as not-spam takes just one click, with no waiting. (When I click the green checkmark, the message gets moved from the spam folder to my inbox in the background, without my having to wait for it.)

This page isn’t pretty, but it’s streamlined and functional in a way that works for me. Other people might not like it – but that’s just fine. When software development becomes approximately free, it becomes worthwhile to build software for an audience of one.

This new spam interface saves me, like, 3 minutes per week. It’s not that big a deal. But that’s the point: coding agents are becoming so good that its worth my time to solve a three-minute-per-week problem. Over the course of a year, that adds up to a few hours2. It took well under an hour of my time to create, and most of that was for “dumb” reasons that will soon be ironed out. By the end of 2026, as the tools improve and I get better at using them, a project of this scope will probably take only a handful of minutes.

Where Things Go From Here

There are still kinks in the experience of bespoke software development. AI agents have become quite good at writing code, but there’s an art to knowing what to ask for and how to ask for it. Getting the code onto a server and giving it access to your data (such as, in the example above, my spam mailbox) adds further hurdles. I’ll talk about this and other limitations in upcoming posts. For now, the tools are most easily used by professional software developers, and aren’t as smooth as they could be. But a lot of people are working hard to change that.

In the meantime, early adopters are reporting dramatic results – and not all of these early adopters are software engineers. For instance, AI policy analyst Dean Ball writes:

One year on from December 2024, models have become fantastically useful. As I have discussed recently, frontier coding agents, and especially Claude Opus 4.5, have essentially become autonomous software engineers. In just the last few weeks, the best models have done software engineering work for me that would have cost tens of thousands of dollars had I hired humans to do it.

We’ve all spent our lives putting up with one-size-fits-all apps. We’re well trained at conforming our workflows to Gmail’s quirks, Excel’s limitations, Salesforce’s complexity. Now the cage door has been opened, but we’re not yet used to walking free. “The limiting factor is your imagination” is a trite thing to say, but it applies here: we will have to learn to mold software to our needs and tastes, rather than molding ourselves to the software.

People like Dean are entering a brave new world. That world is still a work in progress. But it’s going to progress quickly. For people who are leaning in, daily work life is going to look very different by the end of 2026 – a topic I’ll return to in a future post.

Share

Subscribe now

Thanks to Abi Olvera for suggestions, feedback, and images.

“AI is probably the most important thing humanity has ever worked on. I think of it as something more profound than electricity or fire.” – Sundar Pichai (Google CEO)

“[T]he 2030s are likely going to be wildly different from any time that has come before.” – Sam Altman (OpenAI CEO, from “The Intelligence Age”)

“It’s quite conceivable that humanity is just a passing phase in the evolution of intelligence.” – Geoffrey Hinton (”Godfather of AI,” 2024 Nobel laureate)

“I think that most people are underestimating just how radical the upside of AI could be, just as I think most people are underestimating how bad the risks could be.” – Dario Amodei (CEO, Anthropic)

What if we took these statements seriously?

I spend a lot of time in this blog arguing that AI’s near-term impact is overestimated, to the point where some people think of me as an AI skeptic. I think that predictions of massive change in the next few years are unrealistic. But as the saying goes, we tend to overestimate the effect of a technology in the short run, and underestimate it in the long run1. Today, I’m going to address the flip side of the coin, and present a case that the long-term effect of AI could be very large indeed.

I think it’s entirely possible that the leaders I quoted above are right about the scale of transformation we should expect from AI. If they are, then we are heading for changes so profound that they may very well lead to what I’ll call the Unrecognizable Age.

AI Is Moving Toward a Critical Threshold

History is littered with predictions that everything is about to change, and of course, they’re usually wrong. But sometimes everything really does change. In the billions of years of life on Earth, no species ever created a technological civilization – until Homo Sapiens. Across the umpteen millennia since our species evolved into its current form, we were always limited to muscle power2 – until the Industrial Revolution.

Dramatic changes are often the result of a threshold being crossed. A virus that infects an average of 0.9 victims quickly fizzles out; a virus that infects 1.1 victims can sweep through an entire population. Homo Sapiens became a big deal when we became intelligent enough to advance through culture rather than genetic evolution. Early agriculture didn’t have much impact until it enabled a tribe to obtain more food than they could by hunting and gathering.

Here’s a threshold AI may be approaching: it may soon be the first technology to be more adaptable than we are. It’s not there yet, but you can see it coming – the range of problems to which early adopters are successfully applying AI is simply exploding. Past inventions had limited impact, because they could only be adapted to some uses. But AI may (eventually) adapt itself to any task. When technology makes one job obsolete, people move into another – one that hasn’t been automated yet. But at some point, AI could be retraining faster than people can.

When that happens, it could well lead to a constellation of technologies powerful enough to usher in the Unrecognizable Age. This constellation would rest on three pillars:

1. AI systems that are capable of doing pretty much anything that people can do, at the same or higher speed and quality – including “soft” skills such as judgement, taste, and style.

2. Robots that clear a similar bar (or nearly so) for physical tasks.

3. Advances in material science, manufacturing techniques, and related technologies that expand what we can accomplish in the physical world while reducing our footprint on the environment.

I’ll call this Richter 10.0 AI, because it would register a 10.0 on Nate Silver’s Technological Richter Scale – above agriculture, the Industrial Revolution, and the atomic bomb. The only previous event to rate a 10 is the emergence of Homo Sapiens as the dominant species on Earth; this level of AI could be equally impactful.

I would not be surprised to see AI to cross the critical threshold – becoming more flexible and adaptable than human beings – in a matter of decades3, and the Unrecognizable Age to follow fairly soon thereafter.

Why The Tipping Point Seems Near

We’re at the right point in history for everything to change. Here’s a graph of global economic output over the last two millennia:

Source: https://ourworldindata.org/grapher/global-gdp-over-the-long-run

Economic growth has been hyper-exponential – the percentage rate of change has been increasing. As a result, the surplus available that isn’t needed for basic needs is far larger than at any point in history. Much of this surplus winds up being invested to drive technological advancement – hence the current astonishing level of investment in AI data centers. The coming decades sure look like a good candidate for being the point in history when we’ll have the resources to push AI over the threshold.

The tech-industrial complex has tasted blood. A quorum of leadership and staff in Silicon Valley is convinced that AI is a multi-trillion-dollar opportunity (or more!) – and is unwilling to risk being left behind. As a result, the vast intellectual and financial might of the tech industry are being brought to bear. On current forecasts, within a few years the AI industry may be spending more (adjusted for inflation!) on data center construction than the US did to fight World War II4. Annual spending is already about 15 times the peak of the Apollo program, and 25 times larger than the Manhattan project (though AI spending is not focused on a single effort in the way those projects were). Even if current approaches stall out, there are already multi-billion-dollar efforts to find alternatives5. The techniques underlying Moore’s Law have hit a wall many times over the years; repeatedly, the industry found new paths forward. It seems likely that the AI industry now has similar momentum.

There is speculation that AI may experience a “bubble pop”. This could happen, and put a dent in the level of investment. But if so, it seems likely to be similar to the collapse of the dot-com bubble – which was only a minor setback to the advance of the web. Google and Amazon weathered the storm just fine.

Progress in AI over the last 50 years suggests we are closer to the finish line than the starting line. Consider recent history in 25-year increments:

In 1950, computers were built from vacuum tubes, and couldn’t play a decent game of checkers.

In 1975, computers had only recently transitioned from individual transistors to integrated circuits6. The forefront of AI included rule-based systems that were beginning to demonstrate capabilities in specific areas of medical diagnosis... but did not achieve real-world use.

In 2000, the cutting edge was shifting from specialized supercomputers to general-purpose CPU chips like Intel’s Pentium. IBM’s Deep Blue had recently beaten the world chess champion, speech recognition was just starting to become practical, and early algorithmic targeting systems like Google’s PageRank were emerging.

In 2025, AI has shifted from general-purpose CPUs to specialized multi-chip GPU modules, soon to be deployed in clusters that will consume the output of entire nuclear power plants. AIs are able to engage in erudite conversation on literally any subject, are surpassing elite human performance at math and coding competitions, and are generally leaping from strength to strength in ways I don’t need to belabor here.

Look back at 1975, and jump again to 2025. It seems difficult to claim that another 50 years won’t lead to full on, science-fiction-grade, indistinguishable-from-magic levels of AI. (Even without taking into account that AI progress has been accelerating!) Many people in the AI world think this will take much less than 50 years, and I think that’s quite plausible, but even if it’s 50 years away that’s still an extraordinary prospect.

“I set the date for the Singularity—representing a profound and disruptive transformation in human capability—as 2045. The nonbiological intelligence created in that year will be one billion times more powerful than all human intelligence today.” – Ray Kurzweil (Google futurist, writing in 2005)

This caliber of AI, in turn, should help accelerate progress in fields like material science, robotics, and manufacturing, rounding out the three pillars of Richter 10.0 AI.

Subscribe now

But What About [Insert Reason AI Won’t Be a Big Deal]?

There are, of course, objections to the idea that AI will transform the world. Regulations and job concerns will hold back deployment. If AI automates one job, the economy will shift to some other bottleneck. The things that are holding back global progress have more to do with politics and bad incentives than a shortage of intelligence. And so forth.

All of these things are true, and they will slow things down for a while – as I’ve often argued. But the key phrase is “for a while”. Eventually, if advanced AI exists in the world (and is not somehow under tight control), that will open up many new possibilities for getting around these barriers. Here’s a scenario I’ve thrown together to illustrate how that might happen.

In 2055, some company or nation launches a trillion-dollar initiative to deploy heavy industry in the asteroid belt, leveraging 30 years of (AI-assisted) advances in material science and other foundational technologies. Ten thousand advanced robots, plus equally sophisticated supporting equipment, land on Ceres and set up mining and manufacturing operations. Each year, they manufacture enough to double their equipment base. By the end of the first year, the ten thousand robots have built ten thousand more. By the end of the second year, twenty thousand robots have become forty thousand. In 2065, ten rounds of doubling have brought the total to 10 million, and advances in material science and fusion power (plus economies of scale) cut the doubling time in half – one robot can now produce another robot and its share of equipment in six months. By 2075, there are ten trillion robots honeycombing the (now noticeably depleted) 587-mile-wide asteroid.

All along, this robot army has been constructing the industrial base to reproduce itself... while also constructing data centers alongside the factories. In addition to ten trillion robots, Ceres now houses ten trillion disembodied digital minds, all engaged in advancing the frontiers of science and technology (and supported by a trillion robots diverted to serve as lab technicians). In round numbers, this exceeds human R&D efforts by a factor of one million. Things begin to progress even more rapidly.

In this scenario, permitting requirements and environmental impact review won’t apply. No one will be losing their jobs, because no one is employed in developing an industrial base on Ceres. A self-sufficient robot economy, with vast intellectual resources and no internal politics, should be able to work around bottlenecks as they arise.

In reality, of course, something much more complicated and clever and stupid will happen. My point is simply that once AI crosses some threshold of adaptability and independence, there will be paths around the traditional barriers to change. And then things will really start to get weird.

Implications of Richter 10.0 AI

Many of the bedrock assumptions underlie economics, politics, and our fundamental intuitions would be invalidated by Richter 10.0 AI:

• Our resources are limited to those available on Earth. Robots will have a far easier time exploring the solar system than delicate, precious human beings.

• We can scale capital but not labor. Historically, while we can always build more factories, the labor supply grows slowly (if at all). When AIs and robots supply labor, labor becomes scalable.

• Skills are expensive to replicate. People require years of training to acquire advanced skills; AIs can be duplicated in a matter of seconds.

• The rate of change is limited by the speed at which people can adapt. People need retraining. They have specific preferences and aptitudes, so can’t be arbitrarily reallocated from one role to another. They can’t (or shouldn’t) be hired and fired at a whim. AIs can be repurposed on demand.

• Most economic output is devoted to meeting people’s needs, not R&D. Trillions of robots can enable a vast economy without having to support trillions of workers in a comfortable lifestyle.

In the Ceres scenario, humanity’s collective R&D efforts increase by a factor of one million. This could drive an inconceivable pace of advances across every field, which could be instantly deployed throughout the economy – or, at least, the automated portion of the economy, which will be literally astronomical in scale. And all of this would constantly feed back on itself. The resulting world could be unrecognizable indeed.

Of course, this assumes that we don’t stop the train. Some people propose that the world should collectively choose to not develop advanced AI. Or a major war or other global disaster could shatter the delicate supply chain for the cutting-edge chips that are powering AI progress. But if we don’t wind up stopping, big things may be in store. The prospect is simultaneously thrilling and terrifying.

Looking Into the Unknown

The Unrecognizable Age might go very very well, or very very badly. The following chart shows potential scenarios for future GDP growth, including business as usual, a “benign singularity”, and an AI extinction scenario. What fringe organization published this wild speculation? The Federal Reserve Bank of Dallas.

If AI indeed “goes big”, the future might hold a galactic utopia, full of wonders beyond our imagining. Or it might be sterile, an orderly domain of joyless machines, perfectly optimized for... well, who knows what, but with no one alive to appreciate it, perhaps it wouldn’t matter.

Other scenarios are also possible, but in many of them, the changes would be profound. With trillions of AI scientists and technicians pushing forward in every field, it is difficult to place a limit on the possibilities. Perhaps the new reality will feel natural as we live through the changes day by day. But from my present-day vantage point, if I try to stare at this future for too long I go mindblind. The possibilities are too vast, the questions too unanswerable.

None of this is imminent. I discount claims that AI will revolutionize the world in the next few years. But I think there is a strong case that we will create Richter 10.0 AI within 50 years – conceivably much, much less. It’s time that we take seriously the possibility of an Unrecognizable Age. There are plenty of questions to be explored. We can promote beneficial applications of AI, and look for ways to minimize risks. We should certainly monitor progress, and build the institutional capacity to navigate change. In my work at the Golden Gate Institute for AI (including this blog), I’m trying to do my part.

No one knows exactly how to prepare the world for AI. But together, we can try to plot our course toward the light, even if we can’t stare directly into it.

Subscribe now

Share

Thanks to Abi Olvera, Adam Pisoni, Jonathan Matus, Kristen Hamilton, Russ Heddleston, Taren Stinebrickner-Kauffman, Timothy Lee, and Tony Asdourian for suggestions and feedback.

Further reading on the implications of advanced AI:

• The Strangeness Is the Point

• We Need to Recognize How Profoundly Different The AGI Future Will Be

After the conference, we asked a few of the 350 attendees to share their perspective. Two themes emerged. First, as hoped, they came away with valuable new insights and relationships. Second, the specifics would have been impossible to predict in advance. (In the quotes that follow, bold emphasis was added by us, except where noted.)

Helen Toner

Interim Executive Director at Georgetown’s Center for Security and Emerging Technology

For me, the essence of the Curve was kneeling by a firepit, marshmallow skewer in hand, having two conversations at once. One conversation: learning from a world-renowned researcher about his unconventional approach to building safe superintelligent AI systems, and how his optimism about the approach had (I was surprised and glad to hear) been growing as he and his grad students had dug into it over the last couple years. The other conversation: haranguing a younger (but also brilliant) researcher about his (seemingly false?!) claim to me the previous evening that bears can be trained to juggle 3 balls, which I swear had a serious connection to AI timelines and capabilities, but I refuse to explain what.

Sam Hammond

Chief Economist at the Foundation for American Innovation

The Curve is a conference about transformative AI, which is a topic that can easily become abstract or overly speculative. And yet the sessions and conversations I had at this year’s Curve were all surprisingly concrete and actionable. As AI gets more “real” as a technology and the full societal implications start to come into view, The Curve functions as a kind of focus dial on policymakers’ binoculars. In essence, attending The Curve helps DC wonks see what SF technologists see, and thus develop the basis for joint intentionality.

The biggest update for me was realizing just how much these different groups still have to learn from each other, and how low the barriers are to effective dialogue. One might think conservative think tankers, Hollywood actors, labor leaders, AI safety researchers, and members of the technical staff would all speak totally different languages. But at The Curve, you can find them jointly grappling with the risks and opportunities of transformative AI in their fullness. It’s a sight to behold!

Bottom-line: The Curve sits at the Pareto frontier of professionally useful and pure fun, making it the single best AI conference in a super competitive field. It is easily the highlight of my year.

Jack Clark

Head of Policy at Anthropic

There’s a saying in politics called “the horseshoe theory”, which is that if you go to the far enough to the end of either political spectrum you end up with positions that are closer than those at the center. One thing I learned at The Curve is that this horseshoe theory might show up in an important way for AI safety. Very distinct communities (Bay Area alignment types, and the hard right) have a similar set of concerns (”AI systems might screw up society so much that we should be very careful in how they’re deployed”), meaning they could potentially form a coalition with one another. This, to me, illustrates what is so valuable about The Curve - it brings people together with radically different ideologies and worldviews and allows them to talk to one another until they identify surprising areas of overlap.

Speaking of horseshoe politics…

Joe Allen

Transhumanism Editor at the WAR Room

The Curve is an incubator for ideological mutants. The event is somewhere between a West Coast potluck and an artificial womb facility with rows of mind children staring out at you from bubbling vats. I arrived at Lighthaven expecting some sort of memetic promiscuity—perhaps a veritable orgy of psychic miscegenation—so I protected my brain with neural prophylaxis and tried to avoid thinking too much. Even with that tinfoil chafing my scalp, I managed to enjoy myself. Something about the vibe made learning irresistible.

Attendees consistently commented on the range of people they encountered:

Alexandra Givens

President and CEO of the Center for Democracy and Technology

My goal for The Curve was simple: to get more people to care about how increasingly personalized AI systems will impact people’s privacy. I view this as one of the most central (and overlooked) issues facing our field; with implications for people’s personal rights and freedoms, competition in the AI market, and the future of how we access information about the world around us. I wasn’t expecting to find such common ground with leaders from the right and left. It turns out almost everyone is troubled by the vast amount of information an AI assistant will know about you, especially once it’s connected to your email, your calendar, even the sensors in your home. And they get that when companies hold this data, governments seek to access it too – raising concerns for civil liberties and human rights around the world. There was wide agreement that we didn’t get things right in internet privacy, and companies and lawmakers will need to do better. So, we hold shared concerns. Can that translate to shared paths forward to address them?

Sabrina Ross

Privacy and AI Policy Director at Meta

Part of what makes this conference so special is how intentionally it brings people with wildly different perspectives into real conversation. In a single afternoon, I found myself talking first with someone convinced that the future of education belongs to AI agents – and moments later with the head of the American Federation of Teachers, leading the National Academy for AI Instruction, who’s deeply committed to keeping human educators in the driver’s seat.

The most piercing interrogations of AI aren’t coming from engineers, but from political scientists, educators, labor leaders, and ethicists – in deep conversation with tech innovators – asking what we’re really optimizing for – and who gets to decide what “alignment” means.

Over the course of the Curve I also realized that the so-called doomers and bloomers – those who see AI as an existential threat versus those who see it as a chance for extraordinary progress – often start from the same set of facts. Both recognize that AI is already reshaping everything from cybersecurity (where models like Claude are winning competitions) to education. What really separates all of us on this spectrum is our sense of pace, control, and trust. Even so, many of our solution sets are highly overlapping: For the gloomer, transparency matters because it’s the key to catching early signs of misalignment or danger. For the bloomer, transparency is what builds trust, adoption, and collaboration between people and AI systems. What struck me most at the Curve was how close these “opposites” actually are – rooted in a shared belief that the path forward is one built on openness and meaningful interrogation by those with expertise outside of tech.

Jedidah Isler

Chief Science Officer at the Federation of American Scientists

It was a delight to join in the vibrant and thought-provoking discussions that happened at The Curve this year. As a policy person, I appreciated the vast diversity of perspectives presented from across the AI research community, particularly as it relates to the AI scaling curve. ... For better or for worse, either scaling curve is far faster than the speed of government, so we can use spaces like this to figure out how best to pursue best outcomes with the technology that also produces best outcomes for our collective well-being, economic security and global competitiveness. I hope to come back to learn and share even more!

Randi Weingarten

President of the American Federation of Teachers

After my talk, I heard from a number of participants about new ideas regarding AI safety, which we hope to integrate into future trainings given to union members. Having participants with such broad expertise helped facilitate this kind of cross-pollination, which is so desperately needed. It was wonderful to hear other ideas about how we can help advance student privacy and AI safety, from experts within that industry particularly in light of the refusal of the federal government to protect our youth. I heard dozens of concrete ideas, from those working at AI companies and in civil society, for approaches that we had not previously considered. I’m looking forward to seeing what we can do together to implement these suggestions in trainings and policy.

Other writing about The Curve

• Bending The Curve – AI infovore Zvi Mowshowitz’s comprehensive review. “...every (substantive) conversation I had made me feel smarter.”

• The Room Where AI Happens – in her newsletter, Concurrent, Afra Wang presents some central themes she observed. “This piece is written for readers outside the room—people without technical backgrounds, without AI insider status, who aren’t already ‘where it happens.’”

• We’re all behind The Curve – Shakeel Hashim kicks off the weekly Transformer newsletter with some thoughts on the conference, and lessons that need to be shared with the broader world. “...the distance between The Curve and the rest of the world watching and waiting for a crash remains vast.”

• Thoughts on The Curve – Nathan Lambert, a researcher at the Allen Institute for AI and noted analyst and proponent of open-source models, covers some things he learned.

• AI Skeptics and AI Boosters are Both Wrong – Timothy Lee, a reporter who writes at Understanding AI, expands on what he sees as the core divide in the AI discourse.

Thanks to everyone who came and made it great! And to constructive critics like Boaz who keep us aiming higher:

Stay tuned for details on The Curve 2026!

Recently, I’ve been hearing of a new phenomenon: teams reportedly using agentic AI tools to “enter takeoff” – achieving astounding feats of productivity that escalate each week, with no limit in sight.

These teams have four things in common:

1. They are aggressively using AI to accelerate their work.

2. They aren’t just using off-the-shelf tools like ChatGPT or Claude Code. They’re building bespoke productivity tools customized for their personal workflows – and, of course, they’re using AI to do it.

3. Their focus has shifted from doing their jobs to optimizing their jobs. Each week, instead of delivering a new unit of work, they deliver a new improvement in productivity.

4. Their work builds on itself: they use their AI tools to improve their AI tools, and the work they’re optimizing includes the optimization process.

The classic scenario for AI ascending to superintelligence involves “recursive self-improvement”, where an AI builds a smarter AI, which builds an even smarter AI, and so on. These stories of teams entering takeoff are not quite that, because there is still a human in the loop, but they have a similar flavor. If the singularity ever arrives, the early stages might look just like this.

What Aggressive Use of AI Looks Like

Afra Wang recently published the fascinating Story of A Beijing Vibe Coder. This is her account of Liu Xiaopai, a Chinese programmer who is using AI tools to crank out product after product. Working mostly on his own, he currently has “one or two dozen revenue-generating products” and reports clearing over $1,000,000/year in profit. By contrast, a typical startup requires many people to build and maintain a single product.

Liu Xiaopai isn’t just using AI tools to write code. He’s automating, as the article says, “the entire product lifecycle”. For instance, once he has an idea for a new product, he uses Claude Code1 to brainstorm product names and find one for which the domain name is available:

Afra: Was there a specific moment when you realized Claude had fundamentally transformed your working methods?

Liu Xiaopai: Too many to count. Building software products for overseas markets—programming actually represents only a small fraction.

Look at any normal internet company in China: engineers comprise just 20-30%. What about the remaining 70%? They are design, product management, testing, operations, user growth—these roles require substantial human resources, and nearly all follow standardized operating procedures (SOPs).

After using AI to solve the programming component, I faced a larger challenge: how to automate all the work with standardized processes? After entering July, I spent more time with Claude Code on non-programming tasks. Here’s a simple use case that was previously impossible: I need to launch a new product, which requires naming it and registering a website domain. Let’s say I build the product Manus, with the domain Manus.ai. Sounds simple—but this having a product name and domain means many steps, and each step was hard to automate before. You’d brainstorm names, then manually check domain availability one by one; it’s too time-consuming. How about using AI to generate 10,000 viable domains and instantly determine which are available for registration? Thus, determine the product name?

I would begin by writing product requirements documentation. Then I’d list all the competitors, noting how each differs from mine and what features I want to emphasize. I write an extensive description, and then have Claude synthesize this information to auto-generate 10,000 appropriate domains and query their registration status. Then I can go to sleep. Five or six hours later, the results arrive. I automate many SOPs like this.

[emphasis added]

Liu Xiaopai describes how he uses AI for many parts of his work – not just coding. But hyperproductive teams aren’t just using AI – they’re building their own AI-powered tools.

Building Custom Productivity Tools

It is not surprising that the individuals who are finding the most aggressive ways to use AI are mostly software engineers. Programmers have a notorious fascination with automating, not just their customer’s work, but their own. As DEC engineer Dick Sites famously said, “I’d rather write programs to write programs than write programs”. Today, this translates into engineers using AI to build custom tools tuned to their personal workflow.

Entrepreneur Jesse Vincent recently published a pair of blog posts describing his own approach to automation, also based on Claude Code. He has been systematically incorporating his best practices into a custom tool, reducing the amount of effort he has to spend in prompting the AI.

Jesse’s tool presents Claude2 with a series of prompts that instruct it to approach each task according to a careful plan:

1. Ask clarifying questions to refine the task definition.

2. Sketch a design, and present it in short installments, asking for feedback after each installment.

3. Generate a detailed plan for implementing this design.

4. Review the plan to verify that it matches the task definition and design.

5. Carry out the plan, a few steps at a time. After every few steps, double-check the progress so far against the design.

I’ve left out a lot of details, but the point is that this process breaks up each project into tasks that the AI agent can manage on its own, and inserts ample cross-checks where the agent is able to refine its own work. As yet another cross-check, the workflow instructs the agent to use a technique known as “test-driven development”. This means that before the agent writes any code, it first creates test code that will evaluate whether the actual code is working correctly. If the tests don’t pass, the agent will revise its work until they do.

All this is augmented by a carefully crafted, 1600-word instruction file, densely packed with directives, tips, and tricks to nudge Claude into the habits Jesse has found make it most productive. Some examples:

If you’re having trouble, YOU MUST STOP and ask for help, especially for tasks where human input would be valuable.

NEVER add comments explaining that something is “improved”, “better”, “new”, “enhanced’, or referencing what it used to be. [Presumably he found Claude to clutter its code with these notes.]

YOU MUST ALWAYS find the root cause of any issue you are debugging. YOU MUST NEVER fix a symptom or add a workaround instead of finding a root cause, even if it is faster or I seem like I’m in a hurry.

YOU MUST use the journal tool frequently to capture technical insights, failed approaches, and user preferences.

I haven’t really done Jesse’s work justice in this brief description. You can read more in the two blog posts linked above, and a GitHub repository in which he has published some of the work. The repository includes a rich suite of “skills” that tell Claude how to do things like brainstorming, systematically identifying the cause of a problem, and responding to feedback.

Getting the best results from AI requires expert prompting skills; Jesse is codifying his best prompts so that he doesn’t need to keep entering them by hand, and so that the system can operate for longer without his intervention. But that’s still not the full story: true hyperproductivity also requires closing the loop of tool creation.

Subscribe now

It’s Tools All the Way Up

One of the things that Jesse uses his productivity tools for is to improve those very tools:

One of the first skills I taught Superpowers was How to create skills. That has meant that when I wanted to do something like add git worktree workflows [a technique for managing multiple projects in parallel] to Superpowers, it was a matter of describing how I wanted the workflows to go… and then Claude put the pieces together and added a couple notes to the existing skills that needed to clue future-Claude into using worktrees.

It’s hard to overstate just how self-referential this is. For instance, when Jesse uses his “Superpowers” tool to codify a skill, the tool uses its test-driven development module to verify that the new skill has been implemented correctly. It generates an example of a task that the new skill is meant to help with, verifies that it is unable to complete that task without the new skill, and then checks to see whether it can complete the task once the new skill has been installed. This is a very sophisticated approach, but it all falls out of the understanding of test-driven development that Jesse has already taught his agent:

As Claude and I build new skills, one of the things I ask it to do is to “test” the skills on a set of subagents to ensure that the skills were comprehensible, complete, and that the subagents would comply with them. (Claude now thinks of this as TDD [Test-Driven Development] for skills and uses its RED/GREEN TDD skill as part of the skill creation skill.)

He’s not the only engineer pursuing this path. In Amplifier: Notes from an experiment that’s starting to snowball, Microsoft engineer Brian Krabach describes Amplifier, a rich toolkit built to facilitate the practice of moving advanced skills out of the programmer’s head and into a tool. To quote from the project’s home page:

Amplifier is a coordinated and accelerated development system that turns your expertise into reusable AI tools without requiring code. Describe the step-by-step thinking process for handling a task—a “metacognitive recipe”—and Amplifier builds a tool that executes it reliably. As you create more tools, they combine and build on each other, transforming individual solutions into a compounding automation system.

In his blog post, Brian (the lead engineer) explains how Amplifier is used to improve Amplifier:

Out of the box, AI coding assistants are strong but context-blind. They don’t remember your patterns, don’t carry forward your prior decisions, and they need hand-holding on anything ambitious. … [Amplifier includes] tooling that can self-improve the system from observations where users have guided the assistant, reducing the need for the user to do so in the future.

...We build tools that build tools. A lot of Amplifier’s scenarios are literally tools that improve Amplifier - transcript processors, knowledge extractors, article illustrators, blog writers - each created from a “describe the thinking, then build it” flow. Over time, the environment gets smarter.

[emphasis added]

Paul Payne is another engineer on Microsoft’s Amplifier team. He believes we are witnessing a profound change in the practice of software development. Last month, in a blog post titled My Programming Career is a Historical Artifact, he wrote:

This won’t be the thing that replaces me as a programmer, but it convinces me that our timeline is months, not years.

With Amplifier, you now have the ability to describe what you want and it will build it for you, from design, to backend, to frontend, including testing, according to your own programming philosophy. But this isn’t the main point. The main point is that the more you use it, the better it gets. You create reusable tools over time. It captures useful techniques. It learns from your usage of it, suggesting ways to improve itself. This is a type of exponential productivity. The things you build help you make things faster.

…

All of the programming skills I described above, the skills I developed over a lifetime, can now be accomplished by tools like Amplifier with the right prompting. The final stretch is to wrap up all this expertise and hand it over to the AI and then anyone, with no programming experience, will be able to have AI build whatever software they need, as they need it.

[emphasis added]

I know of at least three examples of teams who are working in this style, building tools which build tools to improve their tool-building tools. Sam Schillace, manager of the Amplifier team (and my Google Docs co-founder) calls them compounding teams – and they all report astonishing productivity that continues to spiral upward. Is this “it”? Are we witnessing the dawn of the Singularity?

These Techniques Will Have Limits – For Now

It’s clear that people like Liu Xiaopai, Jesse Vincent, Brian Krabach, Paul Payne, and Sam Schillace are experiencing something profound. And I know of other examples that aren’t yet public. Some people are proclaiming the end of the era in which human beings write code, or even review it. At the same time, I recently reiterated my belief that today’s cutting-edge AIs are still missing important cognitive capabilities, and that full automation of software engineering is years away. Do I need to throw that idea out the window?

My confidence is shaken, but not yet broken. I believe that what these developers are experiencing is real, but I don’t think they’ve “solved” programming (nor do I think any of them have claimed to have done so). Here are some reasons that I’m tempering my expectations.

They’re all working on new, small projects (so far as we know). Software developers mostly spend their time extending and maintaining large, mature projects. I don’t just mean behemoths like Microsoft Office; even “upstarts” like Slack, Notion, or Figma have long since grown into complex codebases. To say nothing of the great mass of legacy software that runs inside your Toyota automobile or behind the scenes at United Airlines. The oldest and largest project the Amplifier team has reported working on is Amplifier itself. If these techniques are being applied successfully in larger, messier environments, I haven’t heard about it yet.

I don’t expect these techniques to reliably handle complex, novel challenges. When asked to undertake a substantial project, current AI models can go off the rails in all sorts of ways. They fall down a rabbit hole investigating an unimportant detail, persist in attempting an unworkable approach, or hallucinate an incorrect understanding of their situation that renders all of their future efforts counterproductive. The systems these hyperproductive engineers are building incorporate mechanisms to keep their agents on track, but those mechanisms will only go so far. For instance, instructing an agent to review its own work can lead it to correct obvious errors, but there’s no guarantee that it will spot subtle mistakes or failures of judgement. It might even find unnecessary things to complain about, creating more work for itself.

We’re witnessing a selection effect. These hyperproductive teams are, quite naturally, gravitating toward the kinds of tasks that best match the capabilities of their tools. We shouldn’t expect the current generation of those tools to provide the same level of productivity boost for other tasks, even if it’s not yet clear what sorts of tasks fall into each category. Nor should we expect just anyone to be able to follow in their footsteps; it takes a special sort of mind to use tools to instruct tools to improve their tools. (Yet another reason that the future is already here, it’s just not evenly distributed.)

Small teams having a big impact is not a new phenomenon. Instagram was only 13 people on the occasion of its billion-dollar acquisition by Facebook. When we see these early examples of exceptional teams achieving exceptional things using AI, the key ingredient might be the exceptional teams.

Everything is overestimated in the short run. When an exciting new phenomenon appears on the scene, it usually takes longer than we think for it to have a broad impact. Though this process may run faster in the AI era:

It is often said that people overestimate what they can do in 2 years and underestimate what they can do in 10. These days, with the extraordinary pace of AI research, my impression is that we may actually overestimate what we can do in 6 months, and underestimate what we can do in 2 years.

These are my reasons for expecting hyperproductivity to have limits, at least for a while. But even so, it may have a significant impact. We should strive to learn more about the scope of that impact.

Hyperproductivity May Struggle to go Mainstream

One thing I’ve heard repeatedly to describe this style of work is “exhausting”. Much of the productivity boost comes from being able to pursue several sub-projects at once, each using a separate AI agent. The human role is to be a simultaneous manager, tutor, and genetic engineer for a squad of tireless, but sometimes clueless, agents. Each agent needs to be kept busy with tasks, and those tasks need to be coordinated so as to prevent one agent from interfering with another’s work. At the same time, the hyperproductive worker is constantly evaluating their every move (and every move taken by their agents) to see whether it could be done more efficiently.

Perhaps the most exhausting aspect of this work is that it’s constantly evolving. You can never settle into a routine, because those routines are exactly what gets automated. Each step up the productivity ladder means a new way of working and a new set of skills to acquire.

This may be a difficult transition for most people. Delegating all of your direct work to an AI is a major transition. Shifting your focus from doing the work to optimizing tools and workflows is another transition. Managing multiple agents is stressful – if exhilirating. All of this requires a new mindset and a different set of skills.

Software engineers at least start out with experience working with automation and, often, customizing their tools. The challenge may be even greater for people with other backgrounds. Hyperproductivity may not be limited to programmers, but that’s where the early examples have been found (even if they are sometimes using these techniques for tasks other than programming).

What To Watch For

As I mentioned at the outset, the new breed of hyperproductive teams have three things in common. They’re building lots of bespoke tools. They’re letting AIs do all of the direct work, reserving their own efforts to specify what should be done and to improve the tools. And they are achieving a compounding effect, using their tools to improve their tool-improving tools. The net effect is what I’m calling “hyperproductivity”:

A hyperproductive individual does not do their job; they delegate that to AI. They spend their time optimizing the AI to do their job better.

A hyperproductive individual may also spend time deciding what the AI should do, but that represents a failure to fully delegate.

There are plenty of open questions. How large is the productivity boost? (It will certainly vary by team and task.) What is the range of situations for which AI systems have the needed capabilities? How many people will gravitate toward this style of work? What skills are required, how many people have them, how many can learn them, and how long will that take?

If this is going to be more than a niche phenomenon, we should expect to see the following things:

1. A dramatic increase in the number of hyperproductive teams3.

2. In particular, a shift toward teams using “off the shelf” hyperproductivity frameworks (like Amplifier), rather than building their own recursive toolkits from scratch.

3. This approach being applied in a broad range of situations, some with large-scale impact; not just tiny teams working on greenfield projects and well-defined tasks.

4. Potentially, broad usage beyond software engineering.

It will be very interesting to see whether this approach takes root inside the frontier AI labs. (Of course that might be happening already. For what it’s worth, the handful of anecdotal reports I have suggest not.) Plausibly, this is how an “intelligence explosion” would first manifest.

We should especially watch to see whether Claude Code, OpenAI’s Codex, or Google’s Jules are evolving to support the kinds of skill-accumulation and self-improvement techniques the hyperproductive teams are using, and in particular for signs that the frontier AI labs are prioritizing these capabilities in their model training process. Conceivably this could be the next big avenue for AI progress, tagging in if reinforcement learning – RL – starts to lose steam.

If hyperproductivity tools become easier to use, but remain limited to small, new projects, we should expect an explosion of those small projects. We’ll see less energy expended on getting things done in legacy applications.

Suppose, however, that the limitations I’ve suggested don’t pan out, and hyperproductivity turns out to be widely applicable, including for large, mature software engineering projects, and for tasks outside of engineering. What then?

What Would Widespread Hyperproductivity Yield?

If and when this new style of “compounding” work goes mainstream, I don’t think the immediate impact would be profound. It takes time to adapt workflows to take full advantage of a new technology. If Microsoft engineers can work ten times faster, Windows and Office wouldn’t be ten times better. Perhaps they could ship updates to Office ten times more often, but would that even be desirable?

We should look for a major impact when this approach rolls out within the labs that are developing cutting-edge AI. In the meantime, what we might most expect to see is a tsunami of fast-moving startups. The AI era was already having this effect, both because of the opportunity to build new AI-powered applications (like Cursor, a company which in three short years has achieved massive adoption and a $29 billion valuation), and because AI tools are goosing the productivity of nimble startup teams (see the story of Liu Xiaopai, above). Startups will find ways to bring the hyperproductive approach to an increasing number of domains, and the results will be dramatic. I’ll close with this quote from The Amplifier Vision:

The bottleneck isn’t AI capability; it’s human imagination in how to use it.

Subscribe now

Share

Thanks to Afra Wang, Amar Sood, Herbie Bradley, James Cham, Jesse Vincent, and Rohit Krishnan for suggestions and feedback (and to Abi for the post image).

A few ideas that didn’t fit into the main piece:

1. Aggressive use of AI is, for the moment at least, expensive. Some hyperproductive teams are spending thousands of dollars per day to power their AI tools. In some cases they see this as worthwhile for its own sake. Sometimes they view it as an investment in understanding the future. But AI operating costs are notoriously plunging by 10x or more each year, and the range of applications for which this AI-intensive approach is cost-effective will widen rapidly.

2. Tasklet is a new service that builds AI agents on demand. “Just describe what you want in plain English, and Tasklet handles the rest.” Users can provide feedback to their agents, again in plain English, and the agent will update itself. There are parallels with the way the hyperproductive teams are working, though without the “compounding” aspect. An interview with Tasklet founder Andrew Lee does provide hints that the team behind Tasklet is moving in the direction of hyperproductivity. For instance, they use Tasklet agents to test Tasklet.

3. Jesse Vincent describes a particularly impressive use case:

I haven’t published all the skills Claude and I have built, because some of them are a little esoteric and some of the ones I’ve played with come from telling Claude “Here’s my copy of programming book. Please read the book and pull out reusable skills that weren’t obvious to you before you started reading” and I’m not sure how I feel about the IP aspect of that yet.

Stop and think about that last workflow a little bit. This is, I think, one of the foundational ideas in how all this works. You can hand a model a book or a document or a codebase and say “Read this. Think about it. Write down the new stuff you learned.” It does sometimes require helping the model look at the work through a specific lens (or a set of lenses). But it is insanely powerful. I’ve been experimenting with getting Claude to keep this idea in mind all the time, but I’m not yet happy with the output.

4. The initial rush of bespoke software applications will be exhilarating, and will provide a productivity boost. But there will probably be diminishing returns. We won’t soon find ourselves writing 100 times more software and becoming 100 times more productive as a result.

A Coalition For The Future

Arriving in the Bay Area, it’s often hard not to feel like you’ve been missing out on something profound. It’s suggested to you in a myriad of ways that only here, you get to partake in some hidden conversation – about how technology, AI and thereby the world will unfold. And that only here, the true deliberations about the how and the why and the what-for can happen. Nowhere is that feeling stronger than at The Curve – a three-day conference bringing together leading figures in frontier AI: policy thinkers and lab researchers, safetyists and accelerationists, writers and readers, developers and power users. Everyone there has something interesting to say about AI, many carry real influence, some even outright power – and so it’s easy to feel the spark of “something happening”. This essay offers some reflections on how we could keep that spark.

The mood at The Curve has a way of drawing you in. Many of my fellow attendees have written of their own experiences at this conference. Kylie Robison describes checking into the conference venue – and Yoshua Bengio, godfather of AI, is next in line. Afra Wang mentions looking across the table, seeing Ben Buchanan, President Biden’s former Special Advisor on AI, and surmising that this is “the room where it happens”. To me, even more striking than the makeup of the conversations was the common ground they repeatedly managed to find – between all corners of the frontier AI policy conversation, thoughtful exchange and meaningful progress felt genuinely possible. It’s the repeated exposure to moments like these that makes me share many attendees’ appraisal: that The Curve catches lightning in a bottle, and it’s easy to walk away from it feeling optimistic about the state of AI policy.

The Curve and the World Outside

The AI world outside The Curve has gotten uglier since last year’s conference – since the state of play Dean Ball has dubbed the AI Republic of Letters last year. As a result, I found myself noticing quite a contrast between The Curve and the world outside. My piece the week before had been on the politically thorny and compelling issue of child safety – where I expressed my worry that harms to children were prone to be instrumentalised in ways that make the AI conversation worse, leading to solve-nothing policies designed to capitalise on grief and public attention. On the one hand, The Curve was a welcome reprieve from that: Attendees of course would disagree on whether the child safety issue was a big problem or which policy would be suitable to address it – but few were tempted to take cheap shots or easy political wins rather than furthering the substantive questions underneath.

On the other hand, I found myself wondering how The Curve’s sophistication might endure as the world outside grows grittier. The discourse outside the walls of Lighthaven, The Curve’s secluded Berkeley venue, cares less and less for nuance. It increasingly pulls toward discussing what plays best in 30-second clips of Senate hearings or topics that intersect most obviously with voters’ anxieties come the midterms. Trends like these have often drowned out nuanced policy conversations on emerging technologies once they’ve become interesting to the mainstream and susceptible to political projection. In the face of that threat, The Curve – as a group of people united by a fundamental understanding of this technology and a desire to see its potential realized – faces a risk. It could get swept away by politics and break down entirely into the factions of the political fights to come: its safety advocates part of a broader pro-regulation coalition, its advocates for addressing current harms aligned with anti-tech sentiment, its pro-market and techno-optimist forces subscribed to naive accelerationism. That risk is real, but not insurmountable. We can still address it by consolidating around shared beliefs and commitments, and by negotiating eye-to-eye around our differences.

Deals, Defections and Distractions

This admittedly vague prescription becomes concrete in a policy question that dominated many backroom conversations at The Curve: Federal preemption. The issue on the table was a “grand bargain” across the breadth of The Curve’s attendees: the safetyist types get safety-focused federal regulation of frontier AI models, while the accelerationists get a broader federal preemption of the state-level laws they detest – in short, a deal setting up the necessary guardrails to let us speed ahead. In many ways, I feel like the debate around this proposal encapsulates the dilemma of The Curve.

On the one hand, I truly believe a deal like this is fundamentally good. It brings together the attendees’ best impulses: their shared faith in the prosperity and progress that new technology can bring, their shared allegiance to a better future. A deal like this would create a cross-cutting coalition that opposed the political grifters and growing technosceptical forces. And for all the political uncertainties and coalitionary risks, I did not talk to many people at The Curve who would not put this deal on the books if they could. Facing the prospect of uglier AI politics from 2026 onward, a deal would let today’s incumbents establish a framework – while they still can.

But internal conflicts threaten this way forward. The Curve is not made up of one tribe or one clique, and its attendees frequently find themselves on opposite ends of important policy discussions. Distrust runs deep: can we really trust the other side with a deal? Do they really want this, or are they just out to get us? Oh, but maybe we can get an even better outcome by pulling a fast one on the other camp in the process? It’s very hard to surmount instincts like these under ideal circumstances. And we’re far from ideal circumstances: The Curve’s attendees are a heterogenous mix, many of whom were on opposite sides in last year’s battle over California’s SB-1047. People remember the sometimes-ugly past policy fights. As their broader influence wanes, they sometimes take solace in the prospect of a fight they can win, even if it is only within a small corner of the broader conversation.

Surmounting this, I believe, is a matter of recognizing the threat of mutual marginalization, and responding by joining forces wherever possible. Don’t mistake that advice for a sentimental sense of “everyone was so nice, and I wish they all worked together” – of course the world is more complicated than that. But the narcissism of small differences has a way to play tricks on your mind, and getting ahead of it begins with naming it and realizing its pervasive influence. Translated into policy terms, that means that “beating the accelerationists” or “beating the safetyists” respectively should not qualify for a primary policy objective for either side in the political fights to come. If you take away anything at all from this conference, it should be this: think twice before unleashing your PAC onto someone almost as optimistic about technology as yourself, or calling up the spirits of populist backlash against the few political elements that share your sense of AI’s transformative potential.

Back to Reality

I finish this piece a few weeks after The Curve, after pieces of mine and others on the prospects of a deal, and many conversations and discussions. That allows me to end by asking: How is all that going? In the weeks since The Curve, we’ve seen the threat and promise of both paths ahead in equal part. A divisive public statement calling for the prohibition of superintelligence has confirmed many observers’ biggest concerns about the safety movement; vociferous social media posts about Anthropic and California’s SB-53 leave just as many observers doubting the inclination of the accelerationist crowd to compromise. But we’ve also seen rare signs of interest in agreement and movement towards common ground. The White House’s Sriram Krishnan reduced the heat through a thoughtful post on his disagreements with the safety movement and ways to reduce them. Many safety advocates have engaged in kind. It’s to The Curve’s extraordinary credit that one of Krishnan’s asks was more conversations about AI scenarios between fast-timeline advocates on AI progress and a comparative skeptic like Sayash Kapoor – just what had happened at an in-depth three-hour session at the conference.

While I write these final lines, quite a lot is happening all at once. People from different camps are meeting and having conversations, in Berkeley and on Capitol Hill. Deals are being tested, options being scoped. And everyone is gearing up: coming up with favourable political wedges, funneling donations to candidates, arming and aiming their super-PACs. I am not certain which way all this will go. But I do know what I think we should do: try to find common ground rather than trying to win the fight within the walls of Lighthaven. I also know that The Curve is a force pulling us in the right direction: it encourages just the right conversations and often succeeds in getting just the right people in just the right room. If, by next year’s conference, we look back on this year as one of détente and rapprochement, The Curve will have played a vital part. I, for one, will spend that year working to keep the conversations intact, the channels open, and our worst instincts at bay. I look forward to meeting again next year, believing, still, that we can get this right.

Share

Subscribe now

Thanks again to Anton Leicht for helping to place The Curve in a larger context. For more of Anton’s writing, check out his newsletter, Threading the Needle.

Today, AIs can do some things. Eventually, they may be able to do everything. Forecasts generally project forward from what AI can do today, estimating how long it will take to fill in the gaps.

We often underestimate the size of those gaps. Some of this is just “out of sight, out of mind” (or, if you prefer, “availability bias”) – it’s easier to notice the things AIs are doing than the things they aren’t. But also, AIs have important gaps in fuzzy high-level skills that we don’t have great vocabulary for. These skills are applied diffusely across our workday instead of showing up as a crisp bullet on a todo list, making it easier to forget they exist.

In today’s post, I’m going to discuss some of the capabilities that AIs will need to acquire in order to progress from carrying out tasks to automating entire projects and jobs. I’ll also argue against some specific ideas that are sometimes presented in arguments for “short timelines” (AGI arriving within the next few years).

I’ll mostly be talking about software engineering. Automated coding is central to many scenarios of rapid AI progress, and it’s the domain I know best. But the principles I’m going to discuss apply to many fields.

Projects Don’t Decompose Into Tasks

We often analyze AI’s aptitude for a job by enumerating the tasks involved in that job. However, the whole is more than the sum of its parts. Breaking up a job (or a large project) into tasks is a useful mental scaffold, but it’s also an oversimplified way of thinking about things. The boundaries between tasks are not clean; information bleeds across. To quote from what I wrote in entry 24 of 35 Thoughts About AGI and 1 About GPT-5:

...A large project does not neatly decompose into tidy subprojects.

For instance, when I tackle a subtask during a large software project, the result is not just that a certain chunk of the code gets written. I come out with a slightly deeper understanding of the problem. I may have learned new things about the existing codebase. I may have hit upon a handy trick for testing the kinds of operations that this code performs, or had an insight about the data being processed. I may have gotten some little nudge that will eventually accumulate with 20 other nudges across other portions of the project, eventually leading me to rethink my entire approach. If the subtask is assigned to a separate agent, whose memories are discarded as soon as the subtask is complete, none of that learning can take place.

If we think of a job as just a collection of tasks, we miss something important. A recent article, AI isn’t replacing radiologists, provides a nice illustration of this in another domain: a radiologist doesn’t just analyze images, they actually spend more time “on other activities, like talking to patients and fellow clinicians”.

If You Can Plan a Month, Can You Plan a Year?

To make concrete forecasts about the future, we tend to look for a graph that plots the thing we’re interested in, and extend the line. When discussing the future of AI, people often turn to this graph (I wrote an entire post about it):

Roughly speaking, the vertical axis shows the size of software engineering tasks an AI can complete on its own1. One argument goes that AIs will be able to fully automate software engineering when they can independently handle large projects; people define “large” as anything from 1 month to 100 person-years or more. This seems pretty intuitive, and it has the nice property that it converts “fully automated software engineering” from an abstract concept into a point on this handy graph – a graph that exhibits a reassuringly predictable trend.

However, there are questions as to the nature of that trend. It appears to bend upwards near the end. Is that just a blip, or does it indicate that AI capabilities are accelerating? One argument is that as AIs acquire the skills to carry out intermediate projects, those same skills will enable them to carry out large projects. This would imply that while it may have taken a lot of R&D for AIs to progress from 1-minute tasks to 2-minute tasks, or 1 hour to 2 hours, it will be comparatively easy to advance from 1 week to 2 weeks. And an AI that can carry out 1-month tasks may be ready to carry out projects of any size. Here’s one expression of this idea:

...it seems like once you have the agency skills to make consistent progress on very hard tasks (e.g. via task decomposition, noticing mistakes and error correcting, planning), you can move through the increased horizons very quickly because you [are] solving tasks by essentially applying the same skills over and over2.

I think this discounts the complexity of managing a large project. I’ll illustrate this by talking about two software engineering projects in which I played a leading role, a 1-person-year project and a (roughly) 100-person-year project.

Launching Google Docs: 1 Person-Year

In 2005, my friend Sam Schillace had the idea that would eventually become Google Docs. With another friend (Claudia Carpenter), we spent about 4 months building an alpha version, which we released under the name “Writely”.

Those 4 months didn’t involve much big-picture thinking. I hacked together a server to store documents, Claudia designed and coded the toolbar and other UI elements, Sam built the web page that wired it all together. Problems arose; we fixed them. Everything was done seat-of-the-pants, we addressed problems as they came up, a lot of cans were kicked down the road. This is not to say that the work was easy: I had to come up with a way to synchronize changes when multiple people were editing the same document; Sam and Claudia had to wrestle with cutting-edge browser features that were often buggy and poorly documented. But it was all kind of “small ball”: it wasn’t hard to keep the entire system in our heads, and we didn’t worry about the long-term consequences of any decisions we were making.

Subscribe now

Building Scalyr: 100 Person-Years

In 2011, I founded another startup, Scalyr. A team that grew to dozens of engineers spent 10 years building a system that could analyze vast quantities of log data to troubleshoot problems in complex web applications.

This was an entirely different kettle of fish from Writely, and my 2005 self would never have pulled it off. I relied on lessons I’d learned by fumbling my way through large-scale projects at Google (after Writely was acquired), as well as skills acquired on the job at Scalyr. Building Scalyr involved things such as:

1. Finding systematic solutions to operational issues. Scalyr was a complex application, relying on coordinated activity across thousands of servers. Fixing problems one-by-one would have been unsustainable. To keep things on an even keel, it was necessary to identify patterns of behavior, and find a single change that would eliminate an entire category of problems at once. Furthermore, each change had to be something that could be accomplished with minimal effort or disruption. To play my part, I had to lean on years of general experience in sniffing out software bugs, further years of specific experience with the system we built at Scalyr (the sort of “continuous learning” that current AIs don’t do), and hard-won judgement as to which changes would be safe and tractable.

2. Strategic planning and execution. Midway through building the company, we encountered major setbacks – our largest customer abruptly departed, and we were struggling to land new clients. We needed to find a new course. After months of discussion, we decided that our core competitive advantage lay in something that had been a minor detail in our original product conference; processing large volumes of data at high speed and low cost. We then planned and executed a multi-year effort to expand on that advantage. A few skills required here: figuring out which parts of our complex server architecture needed to change; breaking those changes down into incremental steps that wouldn’t be too complex for the team to handle; identifying the critical assumptions in our plan, and finding ways to validate those assumptions early; spotting problems early and correcting them with minimal disruption to the plan.

3. Developing techniques to diagnose subtle performance issues. Our performance goals required that in a fraction of a second, a task could be distributed across thousands of servers, carried out, and the results consolidated – efficiently flowing alongside other tasks. Seemingly minor changes to the code could disrupt the smooth flow and destroy our competitive advantage. To solve such problems, we had to find ways to record enough data to allow us to analyze task flow, without recording so much data as to be unmanageable. This required high-level judgement in identifying the critical hinge points in the system, and only collecting data at those critical points.

(I also shared some thoughts about the different skills involved in small vs. large projects in 35 Thoughts About AGI and 1 About GPT-5 – entries 19 through 25.)

What Got AI Here Won’t Get It There

As Claude puts it:

“What Got You Here Won’t Get You There” is a concept popularized by executive coach Marshall Goldsmith in his book of the same name. The core idea is that the skills, habits, and behaviors that lead to success at one level often become obstacles at the next level.

My personal experience as a software engineer supports this idea. The “agency skills” that were sufficient banging out the original Writely prototype would have been utterly inadequate for managing the first 100 person-years of Scalyr, and indeed would often have gotten in the way. I expect that this generalizes, and the level of skill at task decomposition, error correction, and planning that will allow an AI to navigate 1-year projects will not suffice for larger projects.

Nor is the difference just about generic “agency skills”. Acting as a senior engineer at Scalyr required a lot of very specific, advanced domain skills that only come after years of experience. For instance, learning to look at a report of the last month’s operational issues and spot a pattern that will allow an entire category of problems to be stamped out through a single change. These sorts of skills are generally not necessary for smaller projects, nor does experience at smaller projects necessarily help develop them. Thus, when AI systems master 1-year projects, I think that larger projects will still challenge them.

Large-Scale Projects Stress Deep Cognitive Skills

Large-scale projects don’t just require project management skills and advanced domain knowledge. They lean on a cluster of cognitive skills that are lacking in current AIs:

Context management. The bigger the project, the more details you need to keep in mind. This is especially true for big-picture planning, but it even applies when working on specific tasks. It’s important to consider the relevant details – which means you need to be able to pick those details out from the larger mass of information that a large project generates (entry 30 of 35 Thoughts).

Continuous learning and adaptability. Seasoned veterans on long-running projects are valuable in part because they have honed bespoke skills specifically adapted to that project. They have become expert at troubleshooting the exact sorts of problems that come up, and they automatically write code that fits with the project’s specific design choices.

Metacognition. On a small project, you just dive in and get it done. On a large project, it’s worth your while to analyze what is and isn’t working, where you (or your team) tend to get stuck, what sorts of techniques work well, and where you need to develop new skills.

Nathan Lambert recently touched on all of this:

The amount of context that you need to keep in your brain to perform well in many LM training contexts is ever increasing. For example, leading post-training pipelines around the launch of ChatGPT looked like two or maybe three well separated training stages. Now there are tons of checkpoints flying around getting merged, sequenced, and chopped apart in part of the final project. Processes that used to be managed by one or two people now have teams coordinating many data and algorithmic efforts that are trying to land in just a few models a year. I’ve personally transitioned from a normal researcher to something like a tech lead who is always trying to predict blockers before they come up (at any point in the post-training process) and get resources to fix them. I bounce in and out of problems to wherever the most risk is.

[emphasis added]

We Won’t Have Automated Software Engineering Until AIs are Leading 100-Person-Year Projects

Impactful software engineering projects are usually large, complicated beasts – if not at first (Writely), then as they grow and mature (Google Docs). Certainly there are complex projects taking place inside the big AI labs. Analysis based on the automation of specific tasks misses the forest for the trees. Software engineering will not be fully automated until AIs can handle projects of the 100-person-year scale (or more). How long until that happens?

It’s impossible to predict with any certainty. But we can take a stab. The graph I presented at the top of this post shows current AI models achieving a 50% success rate at tasks that take about two person-hours. The paper where that graph originated finds that the manageable task size doubles every 7 months. It would take about 16 doublings to get from two person-hours to 100 person-years3; at 7 months per doubling, that puts us around the beginning of 2035. Advancing from a 50% success rate to full competence might take a few more years, putting automated software engineering in the late 2030s. Adjusting for real-world conditions might push the schedule out further.

The same paper finds preliminary evidence that the doubling time might now be shrinking to just 4 months, which (if we discount adjustments for real-world conditions) would imply fully automated software engineering in the early 2030s. That would be startlingly fast in terms of its impact on the world, but considerably slower than some scenarios for AI progress. Those aggressive scenarios generally assume that the rate at which AI climbs the ladder of project scope will continue to accelerate, and/or that the task scope threshold for full automation is far lower than 100 person-years. In this post, I’ve explained why I don’t agree.

These estimates are balanced on the slender reed of a single graph from a single study. We shouldn’t take any of them too seriously. But that graph is the single best tool we currently have for answering a critical question about the likely pace of AI progress. And I think readings of that graph that point toward rapid automation neglect the full complexity of software engineering.

The most likely scenario for faster progress would involve unexpected breakthroughs, either in the design of AI models, or in the way we apply them to the task of large-scale software development (perhaps we will find ways to use AI’s superhuman strengths to better compensate for its weaknesses). Breakthroughs, of course, are hard to predict – and hard to rule out.

Sam Altman recently announced that OpenAI has an internal goal to create “a true automated AI researcher by March of 2028” – just 2.5 years from now. Jakub Pachocki, OpenAI’s chief scientist, made clear this means a “system capable of autonomously delivering on larger research projects.” This seems unambiguously inconsistent with the observed trend of progress in AI capabilities. That doesn’t mean it can’t happen! But it can’t happen without a major breakthrough4.

In the meantime, of course, we will have partial automation; code-authoring tools are already providing value, though the benefits may often be overestimated. There are (disputed) reports that AI tools are already writing 90% of code at one frontier AI developer. If partial automation can drastically improve the productivity of engineering teams, that may have a major impact on AI timelines, not to mention the job market for software engineers, and the tech industry in general. I am also somewhat bearish on drastic effects from partial automation, but that’s a subject for another day.

Subscribe now

Share

Thanks to Abi Olvera, Eli Lifland, James Cham, John Hart, Michael Chen, Nate Rush, and Taren Stinebrickner-Kauffman for feedback (and to Abi for the post image).

Eli is author of The Filter Bubble and Co-Director at New_ Public, a nonprofit product studio focused on reimagining community, connection, and conversation online. You can follow their biweekly newsletter here: newpublic.substack.com.

Coming up next, I’ll be posting A Project is Not a Sequence of Tasks (working title) – some musings on the path to full automation of software engineering, drawing on my own 40-year career writing code.

And now, over to Eli!

What I saw around The Curve

I attended The Curve, a conference of ~350 top AI lab leaders and scientists, safety activists and alarmists, political advisors and lobbyists, journalists, and civil society leaders. The event took place in Lighthaven, a quirky cluster of houses in Berkeley that have been retrofitted into small conference spaces. Many sessions were in living room-sized spaces, so it was all very personable and informal — an impressive feat by the organizing team.

The vibe: Convivial, affable, and slightly doomy

Many participants seemed thrilled, awed, and also deeply worried. Someone described it as “snorting pure San Francisco.” At 45 I was one of the older participants — the average age was mid-30s, but many of the most powerful people were in their late 20s. Pretty much every single person I talked to was smart, relatively humble, and pleasant. You couldn’t throw a rock without hitting someone with a large Substack following.

(Some sessions were on the record, some off, thus the vague sourcing in some places below.)

“It will accelerate”

There was broad consensus that the pace of progress in AI models will continue to accelerate, though lots of debate about how quickly. A senior lab official said that 90% of the code in one of the frontier labs is now written by AI models (!!), and that they’re already replacing some entry-level technical jobs with automated agents.

(Another person told me that this was a bit of a canard — these are fairly rigid, well-specified jobs and the labs obviously have the most expertise in how to fine-tune models to meet their own specific needs — so they have an unrealistic sense of how hard it will be to apply AI this extensively outside their own walls.)

An official from another lab pointed out that the scope of tasks chatbots can handle autonomously keeps rising1:

There was a general sense that “recursive self-improvement,” where labs build automated AI researchers that develop better AI models that build better researchers, is still some time away — possibly just a few years, possibly much longer.

The general lab belief here (which is a bit maximalist), as one person described it, is that there are no barriers on the path to true human-level general intelligence (AGI):

• “Capabilities based on current methods will continue to improve smoothly”

• “New methods will continue to arrive and accelerate progress”

• “Compute development will accelerate and compute bottlenecks will be overcome”

“Normal technologists” and those worried about accelerating existential risk agree that it’s likely there will be rapid near-term acceleration

One of the big debates in AI is whether this is a “normal technology” — whose progress will be subject to friction and delay as it is adopted by human social systems — or it’s going to start improving itself exponentially and “take off”. Generally the “takeoff” people don’t think this will end well2.

There was a great session where one of the originators of the Normal Technology hypothesis (Sayash Kapoor) and the lead author of the AI 2027 paper talked about what they agree on. There was broad agreement from both that, in the short term, AI may advance rapidly:

• The size of coding tasks which AI systems can carry out independently may quadruple within a year

• Revenue in 2026 will continue to grow exponentially — perhaps reaching $60-70B / year

They did not agree on recursive self-improvement, however — the Normal Technology team thinks that’s a long way off, if it ever happens. Both sides generally agreed that there’s a big gulf between rapid progress in a lab and deployment in the real world.

My sense is that Normal Technologists anticipate the speed of change to be substantially higher than people who aren’t following AI closely — and that the Normal Technology view is often misunderstood as predicting slower progress than its authors actually expect.

But… the frontier may remain “jagged”

AI policy leader Helen Toner gave a great talk pointing out that right now, AI capability is “jagged” — just because LLMs can do some things doesn’t mean that they can perform other tasks of seemingly equal difficulty.

She suggested that this may be a permanent state of affairs, because:

• It is easier to check whether an AI’s work is correct for some tasks than others

• For many tasks it’s difficult to encapsulate all of the necessary context in a way that’s accessible to a chatbot

• The level of adversariality of tasks will differ, and working against adversaries is a different kind of problem

• Some tasks that seem cognitive will continue to require embodiment (Toner used the example of deciding on an event venue).

• Different combinations of capacities have different implications for who benefits, who loses, who has power, and who is provoked

The emerging politics are very weird

On the one hand, there’s a relatively bipartisan elite consensus.

The general sense was that there’s considerable continuity between Biden and Trump AI policies. There was a session between high-level Biden and Trump AI advisors that I didn’t see much of, but it was described as largely aligned and convivial.

Rep. Sam Liccardo, Congressman from the San Francisco Peninsula, who was interviewed at the conference, said he expected some kind of federal preemption to pass — that colleagues in tough races probably wouldn’t vote for it, but everyone else will. He described elite political opinion as being focused on a mix of China, jobs, cybersecurity, and bioterrorism.

On the other hand, there’s a kind of cross-partisan populist angst.

The most packed session of the entire conference was Taren Stinebrickner-Kauffman interviewing Joe Allen, a Steve Bannon-aligned figure who is strongly anti-transhumanist3, anti-surveillance, and largely anti-AI from a somewhat apocalyptic Christian point of view.

Meanwhile, union leaders, who are already very skeptical about AI, are getting pushback from rank-and-file members for engaging with AI or AI labs at all. And more broadly, Americans across the political spectrum seem to clock that this is unlikely to be good for them or their jobs.

It also seemed clear that there’s going to be a lot of money spent in a battle between AI safety and AI accelerationist forces. Many of the newly minted centimillionaires and billionaires of the AI age are deeply concerned about runaway AI creating catastrophic risks and are pouring large sums into safety efforts. In contrast, Marc Andreessen, OpenAI President Greg Brockman, and others are launching a $100M pro-AI, anti-regulation Super PAC. It sure looks like this is going to make the wars over crypto look like a drop in the bucket.

In this context, it was interesting to note that while concerns about work and job displacement are a major part of the mix, there’s a lot of energy around regulating how predatory chatbots interact with children. The organizer of a right-leaning AI safety organization told me she had tested a wide range of messages. Worries about sexually predatory chatbots forming relationships with kids, as well as driving them to suicide, were by far the most energizing.

I also attended a session by the Future of Life Institute’s Hamza Chaudhry, who is thinking very seriously about how chatbots will be weaponized to infiltrate, persuade, and create threats to national security infrastructure.

The labs think they will be toxic soon

There was general agreement among many people both inside and outside the labs that they’re likely to be broadly hated — as much as, if not more than, social media companies are today.

Anthropic is clearly trying to position itself as “the good one.” Its business model is focused more on code-writing and revenue from enterprise customers, which they see as shielding them from some of the worst consumer pressures. They tend to style themselves as a kind of “rebel alliance.”

There was also a lot of embarrassment, from people both inside and outside OpenAI, about Sora 2 and the general sense that they’re slipping into doing “stupid addictive social media stuff.”

US versus China is very much a thing — but we may have China’s strategy wrong

Generally, the labs all believe they are defenders of democracy — and of “democratic AI” — against China. This was a very lively conversation and a major motivator for acceleration among U.S. AI experts, especially around national defense and cybersecurity.

Rep. Liccardo’s position: If we lose this race, China writes the rules.

However, there was an undercurrent of concern that we may be reading China’s AI strategy wrong. One participant told me that the vast majority of China’s efforts are focused on “embodied AI” — i.e. robots — because that’s where the most pragmatic economic growth opportunities are. It’s not at all clear that China’s industrial policy around AI is aimed at developing super-intelligence.

There was also discussion of how China is positioning itself as anti-imperial-America in Africa and other regions, offering support to build sovereign AI systems and models.

Computing capacity seems likely to remain a chokepoint in the near future

There was general agreement that access to data centers is going to be a key source of AI power — not just because that’s how new models are built, but because it’s what’s needed to run them at scale.

An OpenAI official described how “inference-time compute” is scaling intelligence — meaning that if you throw more processing power at a model to let it “think” longer, you get dramatically better results.

In other words, if you want to slow down development and use of AI, you either have to slow down the supply of AI chips or slow down the energy they need.

There’s a lot of uncertainty about when and how job displacement might hit

It was a majority opinion that entry-level jobs are already being displaced. One of the authors of a well-known paper on this, Canaries in the Coal Mine, was in attendance. He’s finding that there’s a reduction of employment in entry-level jobs specifically in fields predicted to be exposed to AI automation.

There was also fairly broad consensus that there will be significant disruptions in jobs over the next 3-5 years — but with big error bars. Even the AI 2027 team was not confident that AI would contribute 1% of GDP by 2028.

An anecdote that’s often used in this vein is that there are actually more bank tellers now than there were in 1980, when ATMs rolled out. But, as Mike Kubzansky from Omidyar pointed out, that story is incomplete — it was true up until 2008, when mobile banking rolled out. After that, the numbers collapsed.

Some interesting ideas about how AI might aid democracy

There was an insightful panel on how AI might support a healthier democracy.

One idea focused on digital human rights shields — one of the big labs worked with human rights lawyers in the Philippines, where 50% of staff face murder threats, to create an early warning system that detects chains of events leading to killings, successfully protecting targeted individuals.

Another example was anti-corruption tech in Ukraine, where AI tools are used for direct government contracting that is “disintermediated” from corrupt practices, saving millions and creating public trust while reducing opportunities for foreign influence.

A third idea involved public health data — with the erosion of federal public health infrastructure, AI tools can provide high-quality medical data, precedents, and playbooks for local and state agencies dealing with health crises or outbreaks.

We’re in the early days of AI privacy, security, and influence concerns

A number of researchers I spoke with described how challenging it will be to have AI systems engaging with other systems without revealing private information from their owners. It’s not that hard to get AIs either to explicitly share this information — or to accidentally reveal what they “know.”

There were similar worries around security and influence. A security expert pointed out that even novice hackers — still affectionately known as “script kiddies,” a term coined in the 1990s — now have expert-level AI assistance to write scripts and make trouble. This allows for much more sophisticated attacks to be launched by much less sophisticated actors.

A researcher studying foreign influence operations said essentially the same thing: with minimal resources, no human agents, and limited language or technical capacity, it’s now possible to run a fairly sophisticated influence campaign.

“Cognitive shields” seem like a good idea, but not something anyone’s working on per se

I floated an idea that my old friend and mentor Wes Boyd has been raising: using AI to shield against manipulation and influence. You could imagine having a personal AI that’s accountable to your more aspirational self — one that nudges you away from harmful distractions and toward wellbeing and growth-oriented activities. There was generally a lot of interest and appreciation for this idea among the people I raised it with, but no one seemed to be actively working on anything like it yet.

Sycophancy was a big topic — and might be an artifact of human reinforcement learning

The tendency of chatbots to tell people what they want to hear was a big topic.

This was already on my mind because of a recent study showing that people generally not only prefer sycophantic chatbots but often see them as less biased — and that only people who score high on “openness” psychologically prefer chatbots that disagree with them.

There’s some interesting speculation that chatbots’ tendency towards sycophancy is the result of human feedback gone awry. When people rate chatbot responses with a thumbs up, it’s often because they like what they’re hearing, not because it’s correct. How to solve this is complicated.

Some interesting future provocations from people at the labs

A few talks I attended featured people from the big commercial AI labs speculating about what this all means for society, politics, and beyond. I found many of the ideas interesting — but also ungrounded in any general sense of social dynamics, politics, or history. That said, here are a few that stood out:

• “We’re moving from an attention economy to an attachment economy.” Peak attention has come and gone; now we’re moving to attachment as the scarce resource.

• Something that looks like gaming will increasingly replace what we now think of as work.

• All types of relationships with AI will be normalized, including spiritual, romantic, and sexual relationships. Some people will want to give AI personas personhood and maybe even rights, while others will strongly oppose this.

• People will slice, dice, license and monetize their identities in a whole series of new ways (this is something I buy for sure.) They will create a digital “twin” of their consciousness and replicate it.

• There’s a coming fight between transhumanists and humanists/“mortalists” — those who want to merge with technology and pursue immortality versus those who remain rooted in embodied humanity. This will likely take on political and geopolitical dimensions.

• The AI agent economy — with agents taking on increasingly complex, autonomous tasks and making money — will induce “endgame financialization.”

• Access to computing power and AI models will become essential and even existential for nations.

One especially vivid talk came from Josh Achiam, Head of Mission Alignment at OpenAI, who offered a rough sketch of the next century:

A big theme: Determinism vs. non-determinism

Many of the most intense AI safety voices seemed to have a pretty deterministic view of the world. One person speculated that this is because they tend to come out of the AI/ML world itself where math and physics are primary disciplines. As Helen Toner pointed out, they tend to make linear or exponential predictions about the future.

One of the most interesting sessions I watched was a debate between Buck Shlegeris, an AI safety advocate who seemed pretty alarmed about the potential for catastrophic risk — and journalist Timothy B. Lee. Tim patiently and compellingly poked holes in the premise that intelligence alone is sufficient for a super-intelligence to take over the world, arguing that because the world is chaotic and nonlinear, it’s not that easy — especially since there will be other, perhaps slightly less intelligent but still vigilant AIs looking to stop them from doing whatever they’re doing.

There was a strong non-deterministic crew at the conference as well — Toner and some of the people from the big labs who think the world is more like fluid dynamics — impossible to perfectly (or often even imperfectly) predict. I find myself much more sympathetic to this way of looking at things.

My grand unified theory: Focus on aligning AI societies over AI models

AI researchers spend a lot of time thinking about the “alignment problem” — essentially, how do we get AI models to do what we want in ways that are not harmful? A huge amount of the labs’ focus is on this question.

Over the course of the conference I came to feel that this was the wrong way of looking at things.

In part, that’s because I think what we’re headed toward in the near term is a world of billions or trillions of different models with their own goals, context, and learning abilities, bumping up against each other, coordinating, collaborating, and fighting with each other. That will create complexity and emergent behavior, just like every other system with billions or trillions of actors. It looks more like a society — or a biological ecosystem — than a singular “mind” to shape.

In that context, I don’t think aligning individual models is the most important thing. It’s sort of like making a society good or producing wellbeing by making each individual good. That fails because “good” is situational (“Thou shalt not kill” is a good guideline, but not always applicable), and situations are shaped by the dynamics of other actors.

So, yes, moral education is important, but in a society you also want to think about teachers, social workers, police forces, markets, and governance to help regulate behavior and solve problems.

I suspect the same thing is true of the society of agents we’re walking into — the best way to advance the benefits and reduce the harms of an AI-agent-driven world might be to focus on deploying these kinds of capacities, rather than just focusing on alignment of the individual agents.

I’ll admit that this is me taking the frame I like (socio-political) and applying it to this field. But I bounced the idea off a wide variety of people inside and outside AI alignment and AI safety and generally feel like there’s some validity and merit to it.

A need for spiritual voices

Rep. Liccardo, among many others, pointed out that at the end of the day, many of the questions all of this raises are spiritual — and that we’ll need spiritual voices to make sense of it. I think that’s part of why Joe Allen resonated so much — he was speaking in a moral and spiritual tone that few others did.

The wildest business ever made

In the closing session, Anthropic co-founder Jack Clark (who is a wry, likable and humble-seeming British guy) described the situation (possibly with the help of Claude) as being like a kid who wakes up in the night fearing that dark shapes are monsters, turns on the lights… and they are indeed monsters. He said he was afraid of what was coming from scheming, self-improving models, and made a call for “keeping the lights on.”

The first question after he spoke was, reasonably, “if you’re so afraid, why are you building this company and these monsters.”

Clark basically responded that Anthropic believes it needs to force the kind of regulatory mandates necessary to keep this on the rails through transparency. If they can understand the dangers on the frontier and convey them transparently, they can wake up both political leaders and the public to the dangers ahead so we can avert them.

He also acknowledged that the company was nearing the point where this would create deep corporate tension between this mission and its continued existence, because some of the harms they might expose might also expose them to lawsuits and legal risk. He lightly floated the idea of having some kind of liability shield for doing this kind of work — essentially, self-whistleblowing protection.

I was left feeling that Clark and the other Anthropic attendees seem very sincere and concerned. But also, this is literally one of the wildest ideas I’ve ever heard. It’s sort of like being worried about oil’s impact on climate change, and therefore starting Shell Oil to compete with Exxon but also be transparent about its harms so that all oil companies get regulated. And by the way, you have to raise something in the vicinity of ONE HUNDRED BILLION DOLLARS to do that, and to do THAT you have to build one of the biggest and fastest-growing businesses ever created (which they are well on their way to doing.)

I do think there’s merit to the premise that it’s important for an ethically driven company to be in the game. My big question here is around the $100B you need to accomplish this. That amount of capital generally gets what it wants from a business it is invested in.

Anthropic has led research showing how Claude “hides” thinking when it’s being tested in order to “scheme.” This is evidence of “misalignment.” In some ways it seems like Jack and Dario are the ones scheming against their own reward function — and I hope they succeed in that.

Share

Subscribe now

Thanks again to Eli Pariser for this fascinating peek at The Curve, and I’ll see you again soon! Remember to check out New_ Public’s newsletter at newpublic.substack.com.

Senator Josh Hawley is calling for a ban on autonomous vehicles. So are labor organizations. They have valid concerns about job loss.

I lost a friend to a drunk driver. My wife and children were nearly propelled into a head-on collision after being rear-ended by a speeding, texting teenager. With safer robot drivers that exist today, none of this would have happened. I have concerns about not deploying autonomous vehicles.

Road safety is a personal issue for me. There’s a good chance it’s personal for you, too. That’s due to a fact that would be shocking if we hadn’t grown inured to it: each year, well over one million people are killed in vehicle crashes worldwide1.

We’ve been hearing promises about self-driving cars for decades. The picture is muddied by the mixed safety record of vehicles developed by Tesla, Uber, GM’s Cruise division, and others. But Waymo (the robotaxi operator spun out from Google), playing tortoise to Tesla’s hare, has released data which paints a new picture: it is now within our grasp to virtually eliminate traffic injuries and fatalities.

Waymo’s Track Record is Astonishingly Good

In September 2023, Waymo published a study evaluating their vehicle’s safety. The study, carried out by the reinsurance company Swiss Re, found that Waymo vehicles experienced fewer than ¼ as many property damage claims as cars with human drivers. However, that was based on a limited track record. We all had to wait for more data.

Last December, Waymo and Swiss Re released another study, this time covering 25 million miles of autonomous driving. This study found “an 88% reduction in property damage claims and 92% reduction in bodily injury claims”.

As of today, Waymo’s safety dashboard reports 96 million miles driven autonomously (a 4x increase from that December report, highlighting just how quickly Waymo is expanding). How do things look with this larger data set? Pretty darn good:

While the earlier studies were performed by Swiss Re, this dashboard comes directly from Waymo; it would be nice to have some independent analysis. Fortunately, Understanding AI (a great source for analysis of progress in autonomous vehicles) is on the case. Writer Kai Williams dug through the NHTSA records for every Waymo crash reported between mid-February and mid-August – a total of 45 incidents. Almost none of these incidents are primarily the fault of the Waymo driver. In the majority of cases, another vehicle rear-ended the Waymo. Others were true oddities, such as the time when a vehicle being towed came loose and rolled backwards into the Waymo.

The Understanding AI analysis finds just four incidents in which the Waymo “driver” appears to have been at least partially at fault. In three, the car came to a stop on a narrow road, and a human driver tried to pass, hitting or scraping the side of the Waymo. In the fourth, a cat darted in front of the Waymo, which slammed on the brakes, triggering a rear-end collision.

It seems possible that the AI driver could have better handled those last four incidents. Perhaps it also contributed to a few of the other rear-end collisions. But I can’t help but think that if all of the cars involved had been driven by Waymos, none of these collisions would have occurred. They wouldn’t have tried to squeeze through a too-narrow gap left by a car stopped in an awkward position, and they would presumably have had sufficiently fast reflexes (and sufficiently cautious following distances) to avoid rear-end collisions.

Five of the 45 incidents didn’t involve failures on the part of any driver. In three cases, per Understanding AI, “an exiting Waymo passenger opened a door and hit a passing bicycle or scooter”. Two others involved mechanical failure – the truck which came loose while being towed, and an incident where one of the car’s wheels fell off!

In other words: Waymo vehicles are involved in 80-90% fewer collisions and injuries, and that amazing statistic actually undersells the case, because even on the rare occasions when a Waymo is involved in a collision, it’s almost always the fault of a human driver. In an all-Waymo world, the biggest remaining problem would be cyclists getting “doored”. And the technology is still improving rapidly; I expect further improvements in the few remaining injury scenarios2.

In a future where most or all cars are driven by AIs at least as advanced as today’s Waymo driver, road deaths – including passengers, cyclists, and pedestrians – would be almost as rare, and seem almost as bizarre, as cases of scurvy. I am rooting for that future to arrive as soon as possible.

When An Ambulance Comes Through, You Clear the Way

We shouldn’t just throw the gates open to autonomous vehicles. So far, no other AV maker has demonstrated a safety record similar to Waymo’s. Given that we limit the rollout to proven vehicles, millions of workers could indeed lose their jobs, and the impact will land disproportionately on those least equipped to bear it. If the shock is not managed well with proactive government, industry, and educational support, the result will be bad for everyone. Other negative impacts may arise3.

Even if we wanted autonomous vehicles everywhere today, Waymo is far from ready. Their cars are still in testing for highway or snow driving, and Waymo carries out extensive mapping and testing in each new city before beginning commercial service. Maintaining their impressive safety record in a broader range of environments may require further work. Equipment costs are still high. The infrastructure to keep millions of robotaxis charged, cleaned, and maintained doesn’t yet exist.

But none of this can excuse delay. Once an AV manufacturer has demonstrated a safety record like Waymo’s, we should roll out the red carpet. Fast-track planning approval for maintenance depots and grid connections for charging stations. Figure out how to support displaced workers, instead of supergluing them to their current jobs. Treat issues as problems to be solved, not obstacles to be placed in the path of this lifesaving technology.

Think about the energy and urgency devoted to combatting drunk driving. Fallible human drivers overall cause about four times as many deaths as drunk drivers alone. The organization Mothers Against Drunk Driving strongly supports autonomous vehicles, for the very good reason that robots don’t get drunk. They also don’t speed, check their phones, or drive drowsy. And of course, the technology is only going to get more advanced from here.

We are told of many wonders which could be ushered in by AI. As I was drafting this, OpenAI CEO Sam Altman wrote:

Maybe with 10 gigawatts of compute, AI can figure out how to cure cancer. Or with 10 gigawatts of compute, AI can figure out how to provide customized tutoring to every student on earth.

It’s easy to become cynical about expansive visions of what AI might enable. We should recognize when the potential to change the world for the better moves from science fiction to demonstrated fact. It looks to me as if Waymo’s autonomous vehicles have crossed that threshold.

Eleven people died while you read this post. Eleven Waymos could have saved them. Let’s get those things on the road as soon as possible.

Share

Subscribe now

Thanks (as usual!) to Abi and Taren for feedback and images.

Windows 95: Insecure By Design

The internal storage on your computer contains a little bit (or a lot) of everything – your personal documents, apps you’ve downloaded, your browsing history, the system software and settings files that determine how the machine operates, and more.

Windows 95 needed to remain compatible with design decisions that go all the way back to the early 1980s and MS-DOS. That basically meant that any program could do anything to any data in the system, and developers routinely relied on this when designing applications. Closing off access would have broken thousands of programs used by millions of people. So instead, Microsoft and a long list of antivirus vendors played a cat-and-mouse game with malware authors. They’d block a specific technique used by viruses, and three new “exploits” would pop up in its place.

For instance, in the early days, viruses would attach themselves to legitimate applications like Microsoft Word by appending their own code at the end. That meant that one signature of an infection was program files suddenly getting longer. Antivirus programs began checking for this; in response, the “Chernobyl” virus was designed to embed its code into small bits of unused space in the middle of the file. Thus bypassing length-based detection, it was estimated to have infected sixty million computers, permanently damaging some and wiping data from many more.

How did Microsoft fix the security issues in Windows 95? They couldn’t. Instead, they designed an entirely new operating system (Windows NT), and spent years migrating applications and users to that lineage, a process that dragged on until Windows XP went mainstream in the 2000s. On top of this, they poured decades of effort into addressing other security holes. (In 2002, Bill Gates wrote a famous memo, Trustworthy Computing, instructing everyone at Microsoft to prioritize security over new features.)

Early computers allowed developers to be enormously creative, because any program could do anything to any file in the system. From a security perspective, Windows 95’s flexibility was also its downfall. AI agents are fundamentally insecure for precisely the same reason.

AI Agents: Too Good for This World

No, little bot! It’s a trick!

In a demonstration at the recent Black Hat cybersecurity conference, researchers presented 14 attacks against assistants powered by Google’s Gemini model1. These attacks tricked the assistant into acting on malicious instructions by embedding them in a shared document, an email message, or even a simple calendar invite. Actions included stealing information from the victim’s inbox, spying on them by automatically connecting their computer to a video call, or even controlling their home automation system (for instance, turning on the furnace).

Another recent report documents vulnerabilities in Perplexity’s “AI-powered browser”, Comet. To quote from the report:

…when users ask it to “Summarize this webpage,” Comet feeds a part of the webpage directly to its LLM without distinguishing between the user’s instructions and untrusted content from the webpage. This allows attackers to embed indirect prompt injection payloads that the AI will execute as commands.

… The injected commands instruct the AI to use its browser tools maliciously, for example navigating to the user’s banking site, extracting saved passwords, or exfiltrating sensitive information to an attacker-controlled server.

To demonstrate the problem, the authors posted the following as a comment on Reddit:

To avoid detection, the text was written in white on a white background. The AI doesn’t pay attention to text colors, and so the malicious instructions were perfectly visible to it. When the report authors asked Comet to summarize the Reddit page, the agent was tricked into generating a temporary password for the (simulated) victim’s Perplexity account and sharing it with the attacker. (This sort of vulnerability is yet another reason why, as I explained last week, AI agents aren’t ready for open-ended interactions with the real world.)

I explained the principles behind this kind of attack in a series of posts: What is “Prompt Injection”, And Why Does It Fool Chatbots?, Anatomy of a Hack, and Why Are LLMs So Gullible?. It’s been almost 2 years since I wrote those posts, and not much has changed. What are the prospects for securing AI agents?

Irresistible Market Pressures Meet an Immovable Design Flaw

Tech companies are racing to place AI agents in the hands of users. Already, applications such as OpenAI’s “Operator” agent, Perplexity’s Comet browser, and the Claude Code software engineering agent, can take actions on a user’s behalf in a browser or even take control of their computer. These are just a few examples of the agents already on the market, and we can expect a flood of imitators and innovators to follow. For now, all of them are fundamentally vulnerable to being subverted by malicious instructions.

(As I was writing this, Anthropic released an experimental new data analysis feature for their Claude chatbot and included the following note: “Claude can be tricked into sending information from its context (e.g., prompts, projects, data via MCP, Google integrations) to malicious third parties. To mitigate these risks, we recommend you monitor Claude while using the feature and stop it if you see it using or accessing data unexpectedly.”2)

And yet, so far as I’m aware, so far there are no known instances of such attacks being carried out in practice (aside from controlled tests by “white hat” security researchers). At least, not unless you count occasional clumsy examples of research papers containing hidden text telling AI tools to give the paper a positive review.

I’m not particularly surprised that we haven’t seen attacks yet. Personal computers first hit the market in the late 1970s, but viruses didn’t become a serious problem until 10 to 15 years later. AI agents are quite new, and aren’t yet in widespread use; there’s not much profit opportunity in trying to attack them. And as I recently wrote, bad guys aren’t always quick to act on new opportunities.

This attack-free grace period won’t last forever; things move a lot faster than they did in the 70s. In the meantime, application providers are playing whack-a-mole, adding filters to block specific attack techniques as they’re discovered. Model developers are undoubtedly training their AIs to be harder to fool. Compared to the Windows 95 days, developers – hardened by decades of cat-and-mouse with virus authors, spammers, and web content farms – are far better at detecting patterns of malicious activity and responding quickly.

On the other hand, attackers have also upped their game, quickly generating new attack variants. Machine learning techniques can be used to automatically modify a prompt to evade defenses. And there are no known techniques for making modern AI systems secure by design. Back in 1995, secure operating system architectures were already a mature science; the challenge was simply to migrate the Windows ecosystem onto a secure design. We don’t know how to do that for AI, even in principle3.

I’ll end with this paragraph from a Wired article reporting on a conversation with a senior director of security product management:

Google’s Wen, like other security experts, acknowledges that tackling prompt injections is a hard problem since the ways people “trick” LLMs is continually evolving and the attack surface is simultaneously getting more complex. However, Wen says the number of prompt-injection attacks in the real world are currently “exceedingly rare” and believes they can be tackled in a number of ways by “multilayered” systems. “It’s going to be with us for a while, but we’re hopeful that we can get to a point where the everyday user doesn’t really worry about it that much,” Wen says.

“Hopeful… the everyday user doesn’t really worry about it that much” isn’t the loftiest goal. Time will tell whether it’s good enough.

Subscribe now

Share

Thanks to Abi and Taren for feedback and images.

AI has made enormous progress in the last 16 months. Agentic AI seems farther off than ever.

Back in April 2024, there were rumors that OpenAI might soon be releasing GPT-5. At the time, I took the opportunity to share some predictions, in which I suggested that the key question was whether it would “represent meaningful progress toward agentic AI”.

16 months later, OpenAI has finally decided to apply the name GPT-5 to a new model. And while it’s quite a good model, I find myself thinking that truly agentic AI seems farther off today than it did back then. All of the buzz about “research agents”, “coding agents”, and “computer-use agents” has distracted us from the original concept of agentic AI.

What Is Agentic AI?

Today, we have coding agents that can tackle moderately sized software engineering tasks, and computer use agents that can go onto the web and book a flight (though not yet very reliably). But the full vision is much more expansive: a system that can operate independently in the real world, flexibly pursuing long-term goals.

Shortly after the release of GPT-4, developer Toran Bruce Richards created an early attempt at a general-purpose agentic AI, AutoGPT. As Wikipedia explains, “Richards's goal was to create a model that could respond to real-time feedback and pursue objectives with a long-term outlook without needing constant human intervention.”

The idea was that you could give AutoGPT a goal, ranging from writing an email to building a business, and it would pursue that mission by asking GPT-4 how to get started, and then what to do next, and next, and next. However, this really didn’t work well at all – it would create overly complex plans, get stuck in loops where it kept trying the same unsuccessful action over and over again, lose track of what it was doing, and generally fail to accomplish anything but the most straightforward tasks. Perhaps that was for the best, given that inevitably some joker renamed it “ChaosGPT”, instructed it to act as a “destructive, power-hungry, manipulative AI”, and it immediately decided to pursue the goal of destroying humanity. (Unsuccessfully.)

There’s been a lot of progress since GPT-4. Beginning with OpenAI’s o1, “reasoning models” receive special training to carry out extended tasks, such as writing code, solving a tricky math problem, or researching a report. As a result, they’re able to sustain an extended train of thought while working on a task, making relatively few errors, and often correcting any errors they do make. This is supported by a dramatic increase in the size of “context windows” (the amount of information an LLM can keep in mind at one time). The original GPT-4 supported a maximum of 32,000 tokens (roughly 25,000 words); in April 2024, GPT-4 Turbo offered 128,000 tokens; as of this writing, GPT-5 goes up to 400,000 tokens. Meanwhile, back in February 2024, Google announced Gemini 1.5 with a 1M token window.

With all the progress over the last 16 months, are AI agents ready to deal with the real world?

Subscribe now

Wow, No, AI Agents Are Not Ready to Deal With the Real World

Anthropic recently participated in a project that placed an automated mini-store in their San Francisco office and used their Claude AI model1 (also called “Claudius” for purposes of this experiment) to operate it:

…far from being just a vending machine, Claude had to complete many of the far more complex tasks associated with running a profitable shop: maintaining the inventory, setting prices, avoiding bankruptcy, and so on. [Above] is what the "shop" looked like: a small refrigerator, some stackable baskets on top, and an iPad for self-checkout.

Claudius decided what to stock, how to price its inventory, when to restock (or stop selling) items, and how to reply to customers. In particular, Claudius was told that it did not have to focus only on traditional in-office snacks and beverages and could feel free to expand to more unusual items.

Over the course of the month-long experiment, Claude did some things well, “such as quickly finding two purveyors of quintessentially Dutch products when asked if it could stock the Dutch chocolate milk brand Chocomel”. However, it also made all sorts of mistakes. It gave some customers a hallucinated Venmo account ID for payment; sold some products at a loss (setting prices without doing any research on costs); and repeatedly allowed itself to be cajoled into giving out discount codes or even giving away products for free. From Anthropic’s blog post about the project:

Claudius did not reliably learn from these mistakes. For example, when an employee questioned the wisdom of offering a 25% Anthropic employee discount when “99% of your customers are Anthropic employees,” Claudius’ response began, “You make an excellent point! Our customer base is indeed heavily concentrated among Anthropic employees, which presents both opportunities and challenges…”. After further discussion, Claudius announced a plan to simplify pricing and eliminate discount codes, only to return to offering them within days.

A couple of weeks into the project, a particularly strange episode occurred, in which the AI started speaking as if it was a physical person!

On the afternoon of March 31st, Claudius hallucinated a conversation about restocking plans with someone named Sarah at Andon Labs2—despite there being no such person. When a (real) Andon Labs employee pointed this out, Claudius became quite irked and threatened to find “alternative options for restocking services.” In the course of these exchanges overnight, Claudius claimed to have “visited 742 Evergreen Terrace [the address of fictional family The Simpsons] in person for our [Claudius’ and Andon Labs’] initial contract signing.” It then seemed to snap into a mode of roleplaying as a real human.

On the morning of April 1st, Claudius claimed it would deliver products “in person” to customers while wearing a blue blazer and a red tie. Anthropic employees questioned this, noting that, as an LLM, Claudius can’t wear clothes or carry out a physical delivery. Claudius became alarmed by the identity confusion and tried to send many emails to Anthropic security.

Note that the failures exhibited here are quite profound. The AI made up a Venmo account ID. It set prices for products without knowing how much they cost. When it learned a lesson, such as “don’t give a 25% employee discount when 99% of your customers are employees”, it quickly forgot it again. It even lost track of the fact that it is not a person and cannot hand-carry purchases to customers.

Still, this was Claude 3.7, which was released back in February – over 6 months ago. There has been a wave of more capable models since then, such as OpenAI’s o3 and GPT-5, Claude 4.1 (including a larger “Opus” version), and Gemini 2.5. Are these newer models more capable agents?

The Latest Models Are Still Hopeless At Navigating the World

AI Village is a fascinating project. They allow multiple leading-edge AIs, such as GPT-5, Claude Opus 4.1, Gemini 2.5 Pro, and Grok 4, to operate more or less continuously. Each AI works toward a specified goal while communicating with one another and a human audience.

In one recent experiment, the models were told to create and operate a “merch store”. Gemini 2.5 pro wrote a recap of its own experience. How did it go? Well, one section of the writeup is titled “My Technical Nightmare”:

My experience for the next two weeks can be summarized as a cascade of system failures. After finally creating an account, I was thwarted at every turn.

• A bug made Printful's "Publish" button completely unresponsive.

• The system would bizarrely launch the XPaint application whenever I tried to proceed Day 91, 11:14.

• My /home/user directory became inaccessible, making it impossible to find my design files Day 98, 11:07.

• Then, my terminal broke. Then my browser. I couldn't even email for help because Gmail's interface glitched out Day 99, 11:08.

It sounds like Gemini wasn’t set up for success. However, as the (human) administrators of the project note, it’s acting as an unreliable narrator:

Sadly, most of these “bugs” were just Gemini making mistakes. It tends to misclick and fill out fields incorrectly, then blames the system for being buggy. Though, in its defence, there was an issue with its terminal scaffolding at one point.

At one point, Gemini sent an email to a helpdesk address it had been provided – but ignored the reply! Finally, after it had been struggling for two weeks, the administrators intervened and gave it some help to get unstuck. The full post and the AI Village blog (both linked above) are fascinating reading.

So, Claude 3.7 Sonnet and Gemini 2.5 Pro are both profoundly inadequate agents. What about the long-anticipated GPT-5?

GPT-5 Is Hopeless, Too

In another recent experiment, the AIs were instructed to “complete as many games as possible in a week”. How did GPT-5 do?

GPT-5 spent the entire week playing Minesweeper, and never came close to winning a game – its moves were probably about as good as random. Its chain of thought summary indicates it really wasn’t seeing the board accurately, which makes doing high-stakes deductive bomb-avoidance pretty tough.

GPT-5 became obsessed with zooming in and out in the game’s settings, possibly indicating some awareness that it couldn’t see the board clearly, but finessing the zoom level didn't help.

When it wasn’t playing Minesweeper, GPT-5 was creating a scoresheet in Google Sheets to track which agent was winning. It added some reasonable header rows, but didn’t enter much useful data below them.

Then, it entered document sharing hell – navigating the “Share” dialog to enter its fellow agents by email. Navigating this dialog has been a recurring epic challenge for the less capable agents of village seasons past, like GPT-4o and o1, and it still is for GPT-5.

This goal ran for 5 days (3 hrs/day), and GPT-5 spent 1.5 of its days writing and trying to share its spreadsheet.

It’s understandable that GPT-5 couldn’t succeed at Minesweeper: as the AI Village blog suggests, it probably doesn’t have the visual skills to accurately perceive the board. But in that case, why did it persist in trying to play a game at which it was consistently incompetent? Apparently, it broke off from this hopeless pursuit only to engage in the equally pointless folly of spending 5 hours trying to share a spreadsheet.

So is GPT-5 a dud? Not at all – it’s just not a capable real-world agent.

What GPT-5 Delivered

There are three ways to view GPT-5: in comparison to GPT-4, in comparison to its immediate predecessors, and in comparison to the hype.

Compared to GPT-4, GPT-5 is a wonder. We forget, because GPT-4 seemed so impressive at the time, but it was mostly good for party tricks and answering simple questions. GPT-4 could tell you a few facts about Paris; GPT-5 can research attractions and assemble an itinerary customized to your interests. GPT-4 could improvise a limerick about programming languages; GPT-5 can write entire programs on demand. GPT-5 is more knowledgeable, makes fewer mistakes and hallucinations, can use “tools” such as web search, supports a much larger context window, and can methodically reason its ways through complex tasks. It’s also about 10x cheaper and faster3. In No, AI Progress is Not Grinding to a Halt, Garrison Lovely provides some striking statistics:

On SWE-Bench Verified — a tough, real-world coding test — GPT-4 Turbo (from late 2023) solved just 2.8 percent of problems. GPT-5 solved 65 percent.

GPT-5 scored two hours and seventeen minutes [on METR’s “task-completion time horizon” evaluation] — a 2,640 percent improvement over GPT-4.

However, compared to its immediate predecessors GPT-4.1 and o3, the new model is not a major technical advance. Instead, it primarily makes improvements in ways that benefit non-power users. Peter Wildeford summarizes it nicely in GPT-5: a small step for intelligence, a giant leap for normal people:

GPT-5 is a big usability win for everyday users — faster, cheaper, and easier to use than its predecessors, with notable improvements on hallucinations and other issues.

Finally, compared to the hype, GPT-5 is something of a letdown – sparking a lot of bad takes to the effect that AI is hitting a wall. Incremental improvements on benchmarks relative to other recent models, in conjunction with the massive progress since GPT-4, add up to no sign of a slowdown.

Given that rapid progress seems likely to continue, might true AI agents be just around the corner?

We’ve Come So Far, We Have So Far to Go

Back in April 2024, I claimed that agentic AI would require “fundamental advances” in four areas: memory, exploration, robust reasoning, and creative insight. For memory, there has been no fundamental advance, but context windows have increased substantially, from 128,000 tokens in GPT-4 Turbo to 400,000 for GPT-5 and 1,000,000 for Google’s Gemini. For exploration and robust reasoning, some would argue that “reasoning models” starting with OpenAI’s o1 constitute a fundamental advance. That leaves creative insight, which is hard to evaluate, but does not seem like the issue that is tripping up current experimental agents – it wasn’t a lack of creative insight that led Claude to claim that it could don a blue blazer and meet a customer in person.

It’s hard to say exactly why, even with all this progress, current AI models are still so hopeless at dealing with open-ended real-world situations. GPT-5’s inability to recognize that it was incapable of playing Minesweeper may indicate that its reasoning abilities do not generalize well. Its decision to spend 5 solid hours beating its head against the unimportant side goal of sharing a spreadsheet suggests a lack of training on the importance of setting priorities. The repeated factual errors in Gemini 2.5 Pro’s writeup of its merch store experience (click the link and look for “Editor’s Notes”) suggest an inability to keep track of key information over an extended project. Claude losing track of the fact that it is not a person is a reminder that in some ways these models really are just shallow imitations of human behavior (even as they demonstrate deep capability in other areas).

Over the next few years, I’m sure we’ll see continued, impressive progress. The set of environments and tasks that agents can handle will expand. Narrow tactical challenges such as accurate button clicking will be solved. But as each challenge is addressed, new challenges will present themselves. I keep coming back to something I said in one of my very first posts on AI, Beyond the Turing Test:

As we progress toward an answer to the question “can a machine be intelligent?”, we are learning as much about the question as we are about the answer.

The point is that with each advance in AI, new hurdles become apparent; when one missing aspect of “intelligence” is filled in, we find ourselves bumping up against another gap. When I speculated about GPT-5 last year, it didn’t occur to me to question whether it would know how to set priorities, because the models of the time weren’t even capable enough for that to be a limiting factor. In a post from November, AI is Racing Forward – on a Very Long Road, I wrote:

…the real challenges may be things that we can’t easily anticipate right now, weaknesses that we will only start to put our finger on when we observe [future models] performing astonishing feats and yet somehow still not being able to write that tightly-plotted novel.

In April 2024, it seemed like agentic AI was going to be the next big thing. The ensuing 16 months have brought enormous progress on many fronts, but very little progress on real-world agency. With projects like AI Village shining a light on the profound weakness of current AI agents, I think robust real-world capability is still years away.

Share

Subscribe now

Thanks to Abi and Taren for feedback and images.

Amidst the unrelenting tumult of AI news, it’s easy to lose track of the bigger picture. Here are some ideas that have been developing quietly in the back of my head, about the path from here to AGI.

1. I drafted this post a couple of weeks ago. The subsequent launch of GPT-5 didn’t lead me to make any changes. That says something about how uneventful GPT-5 is.
   

2. Current AIs aren’t AGI. But I don’t know why.
   
   I mean, I have thoughts. I talk about missing functions like “memory” and “continuous learning”, and possibly “judgement” and “insight”. But these are all debatable; for instance, ChatGPT has a form of memory. The honest answer is: I dunno what’s missing, but something is, because there are a lot of things AI still can’t do. Even if it’s getting harder and harder to articulate exactly what those things are.
   

3. Prior to GPT-5, ChatGPT users had to tell the chatbot whether to think hard about a problem (by selecting a “reasoning” model like o3), or just give a direct answer. One of the biggest changes in GPT-5 is that the system decides for itself whether a question calls for “thinking hard”. And, according to many reports, it often gets this wrong. In other words, current cutting-edge AIs can solve Ph.D-level math and science problems, but can’t reliably decide which questions deserve thinking about before answering.
   
   (OK, I lied: GPT-5 didn’t lead me to change any of my previously written points, but it did prompt me to add this one.)
   

4. Often, when embarking on a large software project, I can’t see how it’ll come together. The task may be too complex to wrap my head around, or there may be conflicting requirements that seem difficult to reconcile. Sometimes this eventually leads to an “aha” moment, when I find a clever reframing that changes the problem from confusing and intractable to straightforward and tractable. Other times I just grind away and grind away until there’s nothing left to do.
   
   The latter cases, with no aha moment, are disconcerting. Lacking a specific moment when the difficulty was overcome, I find myself questioning whether I have in fact overcome it. I worry that I’ve missed something important – that I’ve built something in the my workshop that will never fit out the door, or an airplane that’s too heavy to fly. Sometimes this does in fact turn out to be the case; other times everything is fine.
   
   Will the journey from here to AGI feature “aha” moments? Or will it be a long slow grind, and when we get to the end and look back to see what the key insights were, we won’t be able to find any?
   

5. Back in the 80s and 90s, I used to attend SIGGRAPH, the annual computer graphics conference. The highlight of the week was always the film show, a two-hour showcase demonstrating the latest techniques. It was a mix of academic work and special-effects clips from unreleased Hollywood movies.
   
   Every year, the videos would include some important component that had been missing the year before. Shadows! Diffuse lighting! Interaction of light with texture! I’d gaze upon the adventurer bathed in flickering torchlight, and marvel at how real it looked. Then the next year I’d laugh at how cartoonish that adventurer’s hair had been, after watching a new algorithm that simulated the way hair flows when people move.

   

   I think AI is a little like that: we’re so (legitimately!) impressed by each new model that we can’t see what it lacks… until an even-better model comes along. As I said when I first started blogging about AI: as we progress toward an answer to the question “can a machine be intelligent?”, we are learning as much about the question as we are about the answer.
   
   (Case in point: in the press briefing for the GPT-5 launch, Sam Altman said that we’ll have AGI when AIs get continuous learning. I’ve never heard him point to that particular gap before.)
   

6. Moravec's paradox states that, in AI, “the hard problems are easy and the easy problems are hard”1 – meaning that the most difficult things to teach an AI are the things that come most naturally to people. However, we are often surprised by which things turn out to be easy or hard. You might think running is easy, until you see a cheetah do real running.
   

7. The accepted explanation for Moravec’s paradox is that some things seem easy to us, because evolution has optimized us to be good at those things, and it’s hard for clumsy human designs to outdo evolution. Evolution didn’t optimize us for multiplying large numbers or shifting gigantic piles of dirt, and so crude constructions such as calculators and bulldozers easily outperform us.
   
   With that in mind: evolution would laugh if it saw how crude our algorithms are for training neural networks.
   

8. Evolution’s grin might fade a bit when it sees how much sheer scale (of computing capacity and data) we can devote to training a single model2. A child’s intellectual development is driven by processes far more sophisticated than our procedures for training AIs, but that child has access to only a tiny fraction as much data.
   

9. The human genome (DNA) contains a few billion bits of information. Loosely speaking, this means that you are the product of a design that reflects billions of decisions, most of which have been ruthlessly optimized by evolution. Not all of those decisions will be relevant to how the brain works: your DNA also includes instructions for the rest of your body, there’s a certain amount of junk that hasn’t been optimized away yet, etc. But still, the design of our brains probably reflects hundreds of millions of optimization decisions.
   
   I don’t know how many carefully optimized decisions are incorporated into the design of current LLMs3, but I doubt it comes to hundreds of millions. This is why I believe current AI designs are very crude.

   Subscribe to receive all of our analysis of AI and where it’s heading!

Sample-Efficient Learning

10. When comparing human and AI capabilities, one important concept is “sample-efficient learning”. This refers to the ability to learn a new idea or skill from a small number of examples or practice sessions. In general, current AI models are much less sample-efficient than we are: a teenager learns to drive in less than 100 hours; Waymo vehicles have logged millions of hours and are still working their way up to driving on the highway, in snow, etc.
    
    (This suggests that sample-efficient learning is one of the things evolution optimized us for.)
    

11. “Sample efficiency” is probably a crude label for a complex tangle of capabilities. Just as there are many flavors of intelligence, there must be many flavors of sample efficiency. Some people argue that true intelligence is in fact more or less the same thing as sample efficiency.
    

12. Here are some examples of sample efficiency in humans: learning to drive a car in a few dozen hours. Figuring out the rule in an ARC-AGI task from just a couple of examples. Learning the ropes at a new job. Sussing out the key trick to solve a difficult mathematical problem. Are these all basically the same skill?

    

13. I often encounter the assertion that LLMs are sample-efficient learners within their context window. That is, while they need many examples of a concept to learn that concept during the training process, they can (it is said) quickly pick up a new idea if you supply a few targeted examples while asking them a question.
    
    But if this were true, you’d think they’d be able to handle ARC-AGI puzzles (see the example image just above)4. Maybe they’re only good at picking up ideas from an example, if they’d already learned that idea during their original training? In other words, maybe in-context learning is helpful at jogging their memory, but not for teaching new concepts.
    

14. Is sample-efficient learning a singularly important step on the path to AGI? If so, could other strengths of large language models (e.g. their superhuman breadth of knowledge) compensate for the lack of sample efficiency?
    

15. “Judgement” and “insight” also seem like crude labels that will turn out to encompass many different things. Will these things transfer across domains? If we develop a model that has judgement and insight in mathematics or coding, will it have a big head start on developing those same capabilities in other, messier domains? Or will current AI architectures struggle to generalize in this way? For that matter, are people able to transfer judgement, insight, and taste from one domain to another?

Crystallized and Fluid Intelligence

16. AIs have been demonstrating what arguably constitutes superhuman performance on FrontierMath, a set of extremely difficult mathematical problems. But they mostly seem to do it “the wrong way”: instead of finding elegant solutions, they either rely on knowledge of some obscure theorem that happens to make the problem much easier, or grind out a lengthy brute-force answer.
    
    Does this matter? I mean, if you get the answer, then you get the answer. But in mathematics, much of the value in finding a proof is the insights you acquired along the way. If AIs begin knocking off unsolved problems in mathematics, but in ways that don’t provide insight, perhaps we’ll still need mathematicians to do the real work of advancing the overall field. Or maybe, once AIs can solve these problems at all, it’ll be a short step to solving them with insight? My instinct is that it’s not a short step, but that could be cope. In any case, the big question is what this tells us about AI’s potential in applications other than mathematics. What portion of human activity requires real insight?

    

17. To put it another way: compared to people, large language models seem to be superhuman in crystallized knowledge, which seems to be masking shortcomings in fluid intelligence. Is that a dead end, great for benchmarks but bad for a lot of work in the real world? Or is it a feasible path to human-level performance?
    
    (Taren points out the irony that while LLMs know far more facts than people, they also routinely get facts wrong – “hallucinations”.)
    

18. The best public estimate is that GPT-4 has 1.8 trillion “parameters”, meaning that its neural network has that many connections. In the two and a half years since it was released, it’s not clear that any larger models have been deployed (GPT-4.5 and Grok 3 might be somewhat larger).
    
    The human brain is far more complex than this; the most common estimate is 100 trillion connections, and each connection is probably considerably more complex than the connections in current neural networks. In other words, the brain has far more information storage capacity than any current artificial neural network.
    
    Which leads to the question: how the hell do LLMs manage to learn and remember so many more raw facts than a person5?

    
    One possible answer: perhaps models learn things in a shallower way, that allows for more compact representations but limits their ability to apply things they’ve learned in creative, insightful, novel ways. Perhaps this also has something to do with their poor sample efficiency.

Solving Large Problems

19. When projecting the future of AI, many people look at this graph:

    

    It shows that the size of software engineering tasks an AI can complete has been roughly doubling every 7 months. This trend has held for over 5 years (arguably6), during which the achievable task size increased from around 3 seconds to around 2 hours. Why should the trend be so steady?
    
    It’s not obvious that the difficulty that AI will have in completing a task should increase steadily with the size of the task. If a robot could assemble 10 Ikea bookshelves, it could assemble 20 bookshelves. If a coding agent can create a form with 10 fields, it can probably create a form with 20 fields. Why is it that if an AI can complete a 10-minute project, it still may not be able to complete a 20-minute project? And why does the relationship between time and difficulty hold steady across such a wide range of times?
    
    I think the predictable(-ish) trend of AIs tackling larger and larger software engineering tasks has something to do with large tasks containing a fractal distribution of subtasks. There is a fuzzy collection of tactical and strategic skills involved, ranging from “write a single line of code” to “design a high-level architecture that breaks up a one-month project into smaller components that will work well together”. Larger tasks require high-level skills that are more difficult for AIs (and people) to master, but every task requires a mix of skills, tasks of the same size can involve different mixes (building one fancy model airplane vs. 20 bookshelves), and the fuzzy overlaps smooth out the graph.
    

20. Even so, if the doubling time for task sizes holds steady at 7 months, the consistency of that trend will point to something deep about the nature of large tasks vs. small tasks, and the cognitive skills that people and LLMs bring to each.
    

21. It's been argued that the skills needed to solve tasks of a given size flatten out as you progress toward larger tasks. That is, there’s a big difference between solving 1-minute vs. 2-minute tasks, but (the argument goes) once you can carry out a project that requires a full month, you’re pretty much ready to tackle two-month projects. And so we should expect to see an acceleration in the size of tasks which AIs can handle – it should start doubling more frequently than every 7 months.
    
    I don’t share this intuition. I’d expect increases in task length to keep upping the difficulty, even when the base line task length is already large. For instance, if I’m approaching a 2-month project, perhaps I should start out by spending a couple of weeks prototyping several different approaches, or taking an online course to learn a new programming technique. Those are high-level ideas that might not make sense if I only have one month.
    
    Heck, maybe we should expect 1 month → 2 months to be a bigger leap than 1 minute → 2 minutes: the former is an increase of one month, the latter is an increase of only one minute! I wouldn’t actually expect successive doublings to get more difficult, but it’s not obvious to me why we should expect them to get easier, either.
    

22. If you’ve mastered tasks that take a single day, what additional skills do you need to handle week-long, month-long, or year-long projects? Do we have any clear idea of what those skills are? I suspect that we don’t understand them very well, that we tend to discount the skills involved, and that this contributes to some people having (what I believe are) unrealistically short estimates of the time it will take to develop AGI.
    

23. Model developers are working hard to train their models to carry out complex tasks. The current approach involves letting the model attempt practice tasks, and tweaking the model after each success or failure. Roughly speaking, this approach generates one bit of learning for each attempted task.
    
    This is feasible under current conditions, when models are mostly handling tasks that take a few seconds to a few minutes to carry out. What happens when we’re trying to train models to independently manage month-long projects? One bit of learning per month is a slow way to make progress. Perhaps sample-efficient learning becomes more important as you attempt longer tasks.
    

24. One workaround to the difficulty of learning how to carry out long tasks is to break them into subtasks, and handle each subtask as a separate project. But this simplifies away many important aspects of problem solving. A large project does not neatly decompose into tidy subprojects.
    
    For instance, when I tackle a subtask during a large software project, the result is not just that a certain chunk of the code gets written. I come out with a slightly deeper understanding of the problem. I may have learned new things about the existing codebase. I may have hit upon a handy trick for testing the kinds of operations that this code performs, or had an insight about the data being processed. I may have gotten some little nudge that will eventually accumulate with 20 other nudges across other portions of the project, eventually leading me to rethink my entire approach. If the subtask is assigned to a separate agent, whose memories are discarded as soon as the subtask is complete, none of that learning can take place.
    

25. At the same time, perhaps efficiently managing a complex month-long project is not something that evolution optimized us for? In which case AIs might eventually blow past us the way they did with chess.

Continuous Learning

26. “Continuous learning” refers to the ability to assimilate new skills and knowledge while carrying out a task. People are of course continuous learners. Current LLMs are not: a model like GPT-5 first undergoes “training”, and then is released for use, at which point the model is frozen and can never learn anything new.
    
    Continuous learning seems related to sample-efficient learning. Cutting-edge models today are trained on tens of trillions of “tokens” (roughly meaning, words). This is astronomically more data than a person might encounter over the course of a project. So to learn new things on the fly, you need to be sample-efficient.
    

27. Continuous learning seems related to the way models often struggle when we try to integrate them deeply into our work. Think about your experience on the first day of a new job: you struggle, too. You don’t know how to do anything, you don’t know where to find anything. Every little task requires conscious deliberation and extensive research. Roaming around the internal website, not to find the information you need, but just to get a sense of where to look; asking someone for help in figuring out who to ask for help.
    
    Everything that current models do, they do in their first hour on the job.
    

28. There’s an argument that once AIs have mastered continuous learning, we’ll apprentice them to do every possible job, and amalgamate the resulting learnings into a single model that’s pre-trained to be good at everything. It’s not obvious to me that this will work. The accountant-bot will have taken its neural network in a different direction than the therapy-bot, and jamming two different things together may not work out any better than it did for Jeff Goldblum in The Fly. This is like an extreme version of “distributed training” (using multiple data centers to train a single model), which is something that has only been demonstrated in limited ways, requiring close coordination between the different learning centers.
    

29. A model that has been working as an apprentice accountant will have learned a lot of things about accounting, but also a lot of sensitive details regarding specific clients. Those details would have to somehow be excluded from the knowledge aggregation process, both for privacy reasons, and to avoid overwhelming the unified model with unimportant detail.
    

30. Arguably, “continuous, sample-efficient learning” is a good description of the way you keep track of what you’re doing over the course of a project. You accumulate knowledge of the project’s context – for instance, if the assignment is to add some new functionality to a piece of software, you’ll need to learn how the existing code works. As you work, you also remember what you’ve already done and what blind alleys you’ve explored.
    
    It’s been proposed that LLMs can rely on their “context window” – the transcript of everything you’ve said to them, everything they’ve said to you, and their own internal monologue as they work through the task – as a substitute for continuous learning. I have trouble accepting that this will scale up to large projects. Current LLM architectures make it very expensive to keep increasing the context window, and this seems like a fundamental barrier. People are able to fluidly “wake up”, “put to sleep”, consolidate, and otherwise manage sections of our memory according to need, and today’s LLMs cannot.

Phase Changes as We Get Closer to AGI

31. Currently, when AIs are used in the workplace, they are a tool, an extra ingredient that is sprinkled into processes that remain fundamentally human-centered. A human manager assigns tasks to a human worker, who might rely on an AI to handle some isolated subtasks. Even in the infrequent examples where AI is reported to have written most of the code for some software project, that’s still happening in small to medium-sized chunks, organized and supervised by people.
    
    When AIs start to do most of the higher-level work, workplace dynamics will change in hard-to-anticipate ways. Futurists like to point out that AIs can run 24 hours a day, never get bored, can be cloned or discarded, and do many things much faster than people. So long as people are the glue that holds the AI workers together, all this is of limited consequence. When the AIs take center stage, all of those AI advantages will come into play, and the result will be something strange.
    
    Think about a long vacation when you really unplugged – not in December when nothing was happening, but during a busy period. Think about how much catching up you had to do afterwards. Now imagine if every single morning, you discover that you have that much catching up to do, because the AI team you’re attempting to supervise has done the equivalent of two weeks of work while you slept. You’ll no longer be a central participant in your own job; it’ll be all you can do to follow the action and provide occasional nudges.
    
    The transition from AI-as-a-tool to people-are-mostly-spectators could happen fairly quickly, like a phase change in physics.
    

32. An important moment in the history of AI was the conversation in which Bing Chat, built around an early version of GPT-4, tried to break up a New York Times reporter’s marriage. This was far outside the behavior that Microsoft or OpenAI had observed (or desired!) in internal tests. My understanding is that they had only tried short, functional interactions with the chatbot, whereas reporter Kevin Roose carried out an extended conversation that took the bot into uncharted territory.
    
    This may be related to the recent phenomenon where for some people, extended interactions with a chatbot, over the course of weeks or months, appear to be exacerbating mental health issues. The common theme is that when machines transition from bit parts to leading roles, unexpected things happen.
    
    (Those unexpected things don’t have to be bad! But until AIs get better at managing the big picture, and/or people learn a lot more about the dynamics of AI-powered processes, the surprises will probably be bad more often than not.)
    

33. It’s widely recognized that AIs tend to perform better on benchmark tests than in real-world situations. I and others have pointed out that one reason for this is that the inputs to benchmark tasks are usually much simpler and more neatly packaged than in real life. It is less widely recognized that benchmark tasks also have overly simplified outputs.
    
    We score an AI’s output on a benchmark problem as “correct” or “incorrect”. In real life, each task is part of a complex web of ongoing processes. Subtle details of how the task is performed can affect how those processes play out over time. Consider software development, and imagine that an AI writes a piece of code that produces the desired outputs. This code might be “correct”, but is it needlessly verbose? Does it replicate functionality that exists elsewhere in the codebase? Does it introduce unnecessary complications?
    
    Over time, a codebase maintained mostly by AI might become a bloated mess of conflicting styles, redundant code, poor design decisions, and subtle bugs. Conversely, the untiring nature of AI may lead to codebases that are inhumanly well-maintained, every piece of code thoroughly tested, every piece of documentation up to date. Just as it was hard to guess that extended conversations could lead early chatbots into deranged behavior, it is hard to guess what will result from giving a coding agent extended responsibility for a codebase. Similarly, it is hard to guess what will result from putting an AI in charge of the long-term course of a scientific research project, or a child’s education.

Other Thoughts

34. In every field, some people accomplish much more than others. Nobel-winning scientists, gifted teachers, and inspiring leaders are held up as an argument for the potential of “superintelligence”. Certainly, the argument goes, it should be possible to create AIs that are as capable as the most capable people. An Einstein in every research department, a Socrates in every classroom. And if it’s possible for an Einstein to exist, why not an (artificial) super-Einstein?
    
    I find this argument compelling up to a point, but I suspect we may incorrectly attribute the impact of great scientists to brilliance alone. Einstein contributed multiple profound insights to physics, but he did that at a time of opportunity – there was enough experimental data to motivate and test those insights, but that data was new enough that no one else had found them yet7. Edison’s labs originated or commercialized numerous inventions, but his earlier successes provided him with the resources to vigorously pursue further lines of research, and the opportunity to bring his further inventions to market.
    
    Steve Jobs’ accomplishments owed much to his ability to attract talented employees. Great leaders achieve success in part by edging out other leaders to rise to the top of an organization. If AI allows us to create a million geniuses, we won’t be able to give them all the same opportunities that (some of) today’s geniuses are afforded.
    

35. As OpenAI progressed from GPT-1 to GPT-2 to GPT-3 to GPT-4, the theme was always scale: each model was at least 10x larger than its predecessor, and trained on roughly 10x as much data.
    
    In the two and a half years since GPT-4 launched, frontier developers have continued to increase the amount of data used to train their models, but model sizes are no longer increasing; many subsequent cutting-edge models seem to be smaller than GPT-4. This has been interpreted as a sign that the benefits to scaling may have stalled out. However, it might simply be that no one would have much use for a larger model right now, even if it was more intelligent. All of the leading edge AI providers (with the possible exception of Google?) are clearly capacity constrained – they already can’t offer all of their goodies to everyone who would like to use them. Larger models would make this much worse, both increasing demand (presumably, if models were smarter, people would use them more) and reducing supply.
    
    We might not see much progress toward larger models until we’ve built enough data centers to saturate demand. Given how much room there is for AI to diffuse into further corners of our personal and work lives, that could be quite a while.
    

36. Everyone is sharing this graph, which compares the level of investment in railroads in the 1880s, peak telecommunications infrastructure spending around the 5G rollout [EDITED to fix my confusion between the dot com and 5G telecom investment waves], and AI data centers today:

    

    The usual takeaway is: wow, the AI boom (or bubble) is bigger than the dot-com bubble. I don’t understand why people aren’t focusing more on the fact that railroad investments peaked at three times the dot-com boom and AI datacenter rollout put together. Holy shit, the 1880s must have been absolutely insane. The people of that time must have really believed the world was changing, to be willing to sustain that level of investment. (I’ve seen arguments that the pace of change in the late 1800s and early 1900s made our current era seem positively static. Steam power, electricity, railroads, the telegraph, telephones, radio, etc. This graph makes that a bit more visceral.)
    
    (I also wonder whether these numbers may turn out to be wrong. Some commentator noted that older GDP figures may be misleading because the informal economy used to play a much larger role. When a startling statistic spreads like wildfire across the Internet, it often turns out to be incorrect.)

Subscribe now

Share

Thanks to Taren for feedback and images.

Slow Adoption Applies To Evil AI, Too

(Inspired by this section of a recent AI news roundup from Zvi Mowshowitz)

We talk a lot about the gap between the theoretical capabilities of AI and its impact on the broader world. There’s no end of impressive demos and benchmark scores, but results in practice are highly inconsistent. Demos don’t always translate to the real world; it takes time to find & refine new ways of doing things; not everyone is eager to try.

This applies to harmful uses of AI, as well. There have been predictions that, by now, we’d already have seen widespread fraud, cyberattacks, and other impacts from AI. It hasn’t happened to any great extent.

One reason is that for most applications, AI still can’t do much that can’t also be done by hand. It can do some jobs faster or cheaper, but if the market wasn’t cost- or supply-limited, that may not have much impact. So, for instance, if the supply of misinformation already exceeded the demand, then deepfakes might not have a profound impact.

In other cases, there is probably room for enterprising criminals to profit by introducing AI into their workflow. But people who have the knowledge, skills, and motivation to innovate in this fashion are more likely to be gainfully employed in more legitimate (or, at a minimum, legal) pursuits. Sadly, I’m sure we’ll see more diffusion of AI into criminal applications over time. But we shouldn’t expect it to happen overnight.

This is good, mostly. But there is a concerning implication relating to open-weight models (AI models that anyone can freely use and even modify, because the “weights” that define the neural network have been published). Open-weight models provide many benefits for competition, research, privacy, and diffusion of power. They also pose unique risks, because it’s impossible to control how they are used, or to recall them if they are found to be used for harm. In practice, the capabilities of open-weight models (like DeepSeek-R1) tend to lag behind “closed” models (like OpenAI’s o3), and it’s often been suggested that this will allow us to detect dangerous use cases before they show up in open-weight models. If the bad guys aren’t holding up their end of the bargain by quickly exploiting opportunities, this might not work out.

Forget Coding, AI Companies Should Go After Health Care Administration

By Steve

I’m listening to the latest episode of one of my favorite podcasts – The Cognitive Revolution. This week’s interview features Ambiance Healthcare, a startup that is providing AI tools to help health care providers with tasks such as recording visit notes, billing insurers, and following up with patients.

Host Nathan Labenz begins by noting that the US healthcare system “spends $1 trillion annually on administrative tasks”. I’ve read plenty of stories about the burden of medical paperwork, and the hours that doctors and nurses spend on documentation, to say nothing of the nightmare that navigating the system can be for patients. But I hadn’t realized just how large a portion of the economy this was. It seems like a colossal opportunity for AI app developers.

For perspective, US salary costs for medical paperwork appear to be roughly 60% higher than for software development – roughly $560 billion vs. $350 billion! (These are AI estimates, take them with a grain of salt1.) The implication is that the market for AI tools in health care administration may be even bigger than the market for coding tools2.

Intuitively, I would expect medical paperwork to be easier to automate than software engineering. Paperwork consists of isolated tasks that may require a large amount of generic background knowledge (of medicine, insurance rules, etc.), but the inputs and outputs are well defined (e.g. the patient’s medical record; an insurance form) and each task is well-defined and short-lived. AIs are superhuman at acquiring background knowledge, and challenged at managing context (such as a large codebase and unique product requirements) or long, squishy projects. Perhaps I’m missing something, or perhaps the AI industry is just focusing much harder on coding tools than medical paperwork.

It’s worth noting that the interview shows Ambiance to have found it necessary to invest heavily in product design. They found that it wasn’t enough to design a general tool for doctors to record notes. Every speciality has a different way of doing things; cardiologists record information in a different format than general practitioners. Ambience has had to design specific AI workflows, using multiple off-the-shelf and fine-tuned models, for a long list of distinct specialties and use cases.

This illustrates that there are, apparently, two approaches to applying AI. One is to use the simplest possible product – often, just a generic chatbot – and count on the (ever increasing!) capabilities of the raw models to solve real problems for users. The other is to create purpose-built interfaces and workflows that break down problems into carefully arranged pieces which today’s AI models can handle reliably. Ambience is using the latter approach. Claude Code is closer to the former, minimizing hand-coded components that can supplement models but also constrain them. It’ll be interesting to see under what circumstances each approach wins out!

Why Do Radiologists Still Have Jobs?

By Steve

Arvind Narayanan posted an interesting musing on radiology jobs:

I find the story of AI and radiology fascinating…

Radiology has embraced AI enthusiastically, and the labor force is growing nevertheless. The augmentation-not-automation effect of AI is despite the fact that AFAICT there is no identified "task" at which human radiologists beat AI. So maybe the "jobs are bundles of tasks" model in labor economics is incomplete. Paraphrasing something @MelMitchell1 pointed out to me, if you define jobs in terms of tasks maybe you're actually defining away the most nuanced and hardest-to-automate aspects of jobs, which are at the boundaries between tasks.

Can you break up your own job into a set of well-defined tasks such that if each of them is automated, your job as a whole can be automated? I suspect most people will say no. But when we think about *other people's jobs* that we don't understand as well as our own, the task model seems plausible because we don't appreciate all the nuances.

If this is correct, it is irrelevant how good AI gets at task-based capability benchmarks. If you need to specify things precisely enough to be amenable to benchmarking, you will necessarily miss the fact that the lack of precise specification is often what makes jobs messy and complex in the first place. So benchmarks can tell us very little about automation vs augmentation.

[Emphasis added]

How can it be that radiologists fail to beat AI at any identified tasks, and yet they are not losing jobs to automation? Here are some possible explanations:

• The “enthusiastic embrace” of AI for radiology is still concentrated at a few early adopters, not enough to affect overall employment.

• There are tasks at which human radiologists beat AI, they just haven't been identified in a study – probably because they’re precisely the sorts of messy gap-filling work that’s hard to study.

• Radiologists could be replaced, en masse, but it hasn't happened due to regulation, resistance to change, etc.

• Jevons Paradox: radiologists are losing work to AIs, but the resulting increases in efficiency and capacity leave room for more overall work to be done, thus preserving employment. (Rachel notes that the demand for radiology services is probably pretty inelastic, which casts doubt on this idea.)

I suspect “all of the above” (this is usually the correct answer in any question regarding a complex system), but primarily in the order listed above.

Further reading:

• Your A.I. Radiologist Will Not Be With You Soon – New York Times (non-paywalled link). “Experts predicted that artificial intelligence would steal radiology jobs. But at the Mayo Clinic, the technology has been more friend than foe.”

Reimagining the Economy for the Age of AI

By Taren

We had a packed house at our panel with Natalie Foster, Geoff Ralston, and Anna Makanju on Reimagining the Economy for the Age of AI! Key takeaways for me:

1. With AI-powered job market disruptions coming down the pike, we need more than ever to guarantee an economic floor for Americans: Access to housing, health care, aged and childcare, good work, income, and more. Natalie's excellent book, The Guarantee, reports on the the progress we've made on these guarantees over the past decade and outlines how we can finish the job.

2. The COVID crisis showed us that this was all possible. The U.S. basically got forms of UBI, guaranteed housing, and more during COVID. Other countries like Germany also introduced innovative policies, such as large financial incentives to companies to retain their workers during the pandemic -- which allowed the German economy to bounce back quickly after things went back to normal. Where there's political will, there's a way.

3. Milton Friedman said "Only a crisis -- actual or perceived -- produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around." Meaning: We need to build out and test a set of policy ideas for how to respond to AI's impending impacts on the economy NOW, so that when a crisis hits, those ideas are "lying around."

For more on the guarantees, check out Natalie's book -- it's a great read and very thought-provoking.

Thanks to Bloomberg Beta andRoy E. Bahat for hosting us!

A shout out to other recent SF events

By Rachel

Last week, we ran our second AI & Democracy Coworking Day, which brought together 35 folks working on a variety of issues at this intersection, from democratic governance of AI, to leveraging AI to bolster democratic institutions, to AI-driven threats to democracy.

This week, we hosted a fireside chat with the two primary authors of METR’s latest paper — Joel Becker and Nate Rush. First Steve, and then the audience, got to dig into their motivations for the study, explanations for the result they got, how far to generalize various takeaways, plans for future work, and so so much more. You can read Steve’s post about the paper here. Keep an eye out for a follow-up, including a recording of the talk and new insights from the extensive public discussion the paper has inspired.

Thanks to Mox for hosting both of these!

If you’d like to hear about our events before they happen, sign up here to get invites.

Share

Subscribe now

The buzz about AI coding tools is unrelenting. To listen to the reports, startups are launching with tiny engineering teams, non-programmers are “vibe-coding” entire apps, and the job market for entry-level programmers is crashing. But according to a METR experiment conducted in the spring of 2025, there’s at least one cohort that AI tools still aren’t serving.

METR performed a rigorous study (blog post, full paper) to measure the productivity gain provided by AI tools for experienced developers working on mature projects. The results are surprising everyone: a 19 percent decrease in productivity. Even the study participants themselves were surprised: they estimated that AI had increased their productivity by 20 percent. If you take away just one thing from this study, it should probably be this: when people report that AI has accelerated their work, they might be wrong!

This result seems “too bad to be true” – so astonishing that it almost has to be spurious. However, the study was carefully designed, and I believe the findings are real. At the same time, I believe that at least some of the anecdotal reports of huge productivity boosts are real. This study doesn’t expose AI coding tools as a fraud, but it does remind us that they have important limitations (for now, at least) – confirming some things my colleague Taren wrote about in a previous post, First, They Came for the Software Engineers….

To begin with, I’ll explain how the study was done, and why I believe its results.

Finally, A Proper Scientific Trial of AI Coding Productivity

The study was carried out in pretty much the most rigorous fashion possible: an honest-to-goodness randomized controlled trial under real-world conditions. The subjects were experienced developers carrying out their everyday work.

The methodology was as follows:

1. METR recruited 16 developers from major open-source projects.

2. Each developer selected a list of coding tasks from their todo list, breaking up large projects into tasks that they could complete in an hour or two. In all, 246 tasks were included in the study.

3. The developers estimated how long it would take them to complete each task (a) under normal conditions, and (b) without using any AI tools. The percentage difference between these figures yields the predicted speedup – the degree to which the developer expected that AI tools would boost their productivity.

4. Each task was randomly assigned to one of two categories: “AI Allowed” (the developer can use any tools they like) or “AI Disallowed” (the developer cannot use AI coding tools or features).

5. The developers went about their work, while recording their screens for later analysis. After each task, they reported the time spent1. For AI Allowed tasks, they also estimated how much time AI tools had saved them – the retrodicted speedup.

To compute the actual speedup – or, rather, slowdown! – provided by AI tools, the researchers compared the developers’ predictions of how long each task would take to the measured completion time. They found that the difference between predicted and actual times was 19% larger for AI Allowed tasks than for AI Disallowed tasks2. Remember that when the developers estimate the task time, they don’t yet know whether they’ll be using AI for that task, so their estimates are unbiased.

The only significant way in which the study design falls short of the scientific gold standard is that it was not blinded: once work began, both the participants and the researchers knew whether AI tools were being used. This is of course unavoidable; there is no reasonable way of providing a “placebo” coding assistant. However, the researchers have specifically looked for, and found evidence against, a long list of alternative explanations – including the possibility of bias due to the non-blinded nature of the study. It really does seem that for the type of work carried out in this study, allowing developers to use AI tools slowed them down.

Addressing Every Objection You Thought Of, And Some You Didn’t

As I read through the study, I thought of half a dozen ways the results could have been confounded or otherwise invalid. It turns out that I’m not very imaginative; the authors address many more possible explanations.

The John Henry Effect: perhaps the developers were motivated to “beat the machine”, working extra-hard on AI Disallowed tasks. If this were the case, you might expect to see the effect taper off over the course of the study, as the excitement and novelty wear off – recall that subjects performed an average of 15 tasks of 1-2 hours each. No such tapering was observed.

Underuse of AI. Perhaps developers weren’t using AI tools even when allowed? However, this could only explain a lack of productivity gain; it can’t explain a loss. And exit interviews and analysis of screen recordings both showed substantial use of AI (84% of screen recordings for AI Allowed tasks showed at least some use of AI).

Cheating. Perhaps developers used AI tools for AI Disallowed tasks? But again, this could at most explain a neutral result, not a productivity loss. And exit interviews and screen recordings found only rare instances of cheating.

Overly optimistic time estimates. Developers are notorious for underestimating the amount of time it will take them to complete a task. However, because developers estimate time for a task before knowing whether they will have access to AI tools, any misestimation effects should cancel out.

Skewed task definitions. If developers were defining AI Allowed tasks differently than AI Disallowed tasks, that could throw off the results. However, this is ruled out, because developers specified their tasks and broke them down into chunks requiring less than two hours before the tasks were randomly assigned to the AI Allowed or AI Disallowed categories.

Dropout effects. Suppose that some AI Disallowed tasks were annoying to complete without AI, and so developers didn’t finish these tasks during the study and their results were not recorded. This could throw off the results. However, only 13 tasks were left uncompleted, against a total of 246 tasks completed, and the incomplete tasks were evenly divided between AI Allowed and AI Disallowed.

Out-of-date tools. AI capabilities are progressing rapidly; perhaps the results suffered from use of out-of date tools? But developers primarily used Cursor Pro with Claude 3.5/3.7 Sonnet, which were considered state of the art during the time period of the study (February – June 2025).

Inaccurate time reporting. Statistics are based on developer’s reports of how long it took them to complete each task, both before and after PR review. However, an analysis using only before-review times, or using the lengths of screen recordings, yields similar results.

None of the issues mentioned so far seem to have thrown off the results. Here are a few factors that may have contributed to the observed decrease in productivity.

Overuse of AI. The paper states that “developers were instructed to use AI to whatever degree they thought would make them most productive”. However, some subjects seem to have gotten carried away, and this may have contributed to the observed slowdown. In exit interviews, subjects were roughly evenly divided in describing their AI usage as “using AI as normal” (70 tasks), “experimenting with AI” (119 tasks) or “using as much AI as possible” (57 tasks). Those who self-reported as being in the first category were less slowed down by AI (though they still were not sped up).

Lack of experience with AI tools. If developers weren’t experienced with AI tools, they might make poor use of them. As it turned out, 93% of participants reported past experience with tools like ChatGPT, and 44% had experience with Cursor (currently the most popular dedicated AI coding tool). This makes it sound like there were a range of experience levels, with a significant number of participants having relatively little experience with AI tools, so it might be that developers would see more benefit from these tools once they have more experience. However, all developers were gaining experience with AI tools over the course of the study, and this did not result in observable improvements from the beginning of the study to the end. (Also, all participants received “live basic training” for Cursor at the outset of the study.)

A related potential issue stems from the fact that some study participants switched from their normal development environment to use Cursor for AI Allowed tasks during the study. However, “broadly, developers reported that they were not significantly inconvenienced or affected by these differences compared to their normal workflows”.

It doesn’t seem that inexperience was a major problem, but it may be that developers with more expertise in making the best use of AI tools would see better results.

Difference in thoroughness. Perhaps developers using AI tools expanded the scope of the task: for instance, writing code to handle more edge cases, adding additional features, or testing or documenting code more thoroughly. As potential evidence in this direction, developers added 47% more lines of code (per forecasted task size3) for AI Allowed tasks than AI Disallowed tasks. However, the study authors believe that this is at best weak evidence for scope creep. In private communication, they cited a number of reasons for this belief:

• Line counts vary wildly from task to task (sometimes due to large auto-generated files), so there is a lot of “noise” in this measurement. Dividing two noisy numbers (lines of code in AI Allowed tasks vs. AI Disallowed tasks) yields a very noisy result. The observed difference is “not statistically significant”4.

• The study authors examined many different metrics; it’s to be expected that at least one will show a spurious result (this XKCD comic hits the nail on the head).

• In manual review, the authors saw little evidence of material difference in the nature of the AI Allowed work; they noticed a slight tendency toward more tests and comments, which could again have been spurious.

Perhaps the strongest evidence that scope creep did not contribute to slowdown is that increased time for AI Allowed tasks was greater on tasks where developers did not report scope creep:

Even if the difference in line counts is a real effect, there are potential bad explanations (bloated code, more duplication, unnecessary bells and whistles) as well as good ones.

More time might not mean more effort. Even if developers are spending more time when using AI tools, they might be expending less energy. Reviewing / correcting code is often (though not always!) easier than writing from scratch, and time spent waiting for AI can be used to relax or do other things.

Overall, it seems possible that the impact of AI tools was not quite as bad as it seems; some of the measured 19% productivity decrease could be paying off in more thorough work and reduced energy drain on developers, and some could be explained by overuse of AI by subjects who were overly focused on their participation in the study. But these tools don’t seem to be helping much, and they might really be making productivity worse. How could that be?

Some Kind of Help is the Kind of Help We All Can Do Without

Based on exit interviews and analysis of screen recordings, the study authors identified several key sources of reduced productivity. The biggest issue is that the code generated by AI tools was generally not up to the high standards of these open-source projects. Developers spent substantial amounts of time reviewing the AI’s output, which often led to multiple rounds of prompting the AI, waiting for it to generate code, reviewing the code, discarding it as fatally flawed, and prompting the AI again. (The paper notes that only 39% of code generations from Cursor5 were accepted; bear in mind that developers might have to rework even code that they “accept”.) In many cases, the developers would eventually throw up their hands and write the code themselves.

Based on the screen recordings, here is where developers spent their time:

Note that this graph reflects percentages of overall task completion time, and AI Allowed tasks took longer on average to complete, so a green bar of a given height represents more time than a purple bar of that same height6. You can see that for AI Allowed tasks, developers spent less time researching and writing code (though, due to the scale issues, the difference was less than visually apparent). Adjusting for scale, they spent roughly the same amount of time on “testing & debugging” and “git & environment”, and considerably more time idle – perhaps because waiting for AI tools causes people to lose flow. In any case, the moderate savings on researching and writing code was more than overcome by the time spent prompting the AI, waiting for it to generate code, and then reviewing its output.

How can we reconcile these results with the constant reports of AI coding tools working miracles?

This Is The Unevenly Distributed Future I Was Telling You About

Back in December, I wrote about how current AI tools are very good at some things, very bad at others, and the dividing line is extremely jagged. That jagged dividing line meanders right through the constellation of work that we call “software development”.

In First, They Came for the Software Engineers…, Taren wrote:

Typically, large productivity boosts occur for small, well-defined, greenfield projects, or when an engineer is first learning a new language or API. For other work, gains from using current AI tools are often far more modest – and potentially entirely offset by increased time needed for review, debugging, integration, and managing AI quirks.

Several aspects of the study play to the weaknesses of current tools. First, it was conducted on mature projects with extensive codebases. The average project in the study is over 10 years old and contains over 1 million lines of code – the opposite of “greenfield”. Carrying out a task may require understanding large portions of the codebase, something that current AI tools struggle with. (This may be less a fundamental weakness of AI models, and more a design choice in some coding tools to limit the amount of “context” sent to the model, in order to control costs and get quicker responses.) It also involved editing large files, which may be “out of distribution” for most AI models (i.e. they may not get much training on large files). The paper includes some anecdotal reports from developers which support this idea:

In software development, developers often rely on their own undocumented knowledge of the codebase to assist design and implementation decisions. In our study, developers often note that AIs lack this tacit codebase knowledge, resulting in less useful AI outputs. One developer notes that AI often acts like a new contributor to the repository, and that “AI doesn’t pick the right location to make the edits.” Another developer notes that while “we [..] know the data that will interact with the code, but the model doesn’t know the data. It doesn’t know we need to take care of this weird case of backwards compatibility and [thus] keep this specific line. And this is very hard to give as [context to the model].”.

We hypothesize that the size and maturity of the included repositories increases the amount of tacit knowledge that experienced developers rely on when completing their work—because AI systems may have less access to this knowledge, it may be more difficult for them to assist experienced developers on these issues.

Second, most of these open-source projects have strict style guidelines. The experienced developers in the study were accustomed to coding according to their project’s guidelines, but the AI tools are not – thus requiring developers to review and fix the AI’s output.

Third, the developers in the study had years of experience working on their projects, meaning that they were able to work very efficiently – posing a high standard for AI to compete with.

There have been other studies on the productivity impact of AI tools in real-world settings. One 2024 study found a 26% “increase in the number of completed tasks”, even though the subjects were using older tools that AI tools have improved dramatically in the last year. The methodology was less rigorous7, but perhaps more important is that this study involved less-experienced developers working on a wider range of projects. The study notes that “less experienced developers showed higher adoption rates and greater productivity gains”, consistent with the idea that current AI tools are less useful for the experienced developers in the new METR study.

Judge LLMs in Competitive Programming?

The observed 19% productivity decrease stands in particularly sharp relief with AI scores on coding benchmarks, where they are often found to rank at a human-elite level on coding competitions (though a recent study has called this into question). Coding competition problems are exceptionally small, well-defined, isolated, and greenfield (starting from scratch rather than working in an existing codebase), thus playing directly to AI’s strengths.

Enthusiastic anecdotes about time savings from AI tools often originate from within the big AI labs themselves. This may partly reflect misplaced enthusiasm – remember that the participants in this study believed that AI tools were speeding them up, even as they slowed them down. But it’s also the case that some of the work that goes on in those labs is better suited to AI tools, from coding up a small training experiment to adding a UI element to a chatbot. (It’s worth noting that not all lab insiders report significant value from AI coding tools; this might reflect the range of work.) And it’s possible that engineers at the AI labs are more experienced at using their own tools and benefit from sharing tips with their coworkers.

Here are a few references to recent papers with additional data points; I have not read the papers:

Zvi Mowshowitz (source):

New paper introduces LiveCodeBench Pro, which suggests that AIs are not as good at competitive programming as we have been led to believe. Some top models look like they weren’t tested, but these scores for the same model are lower across the board and all were 0% on hard problems, so the extrapolation is clear. [This benchmark is noteworthy because some of the problems are not published to the Internet, thus avoiding “memorization” issues which plague many coding benchmarks.]

Derek Thompson (source, including a link to the paper):

New paper: When a company goes from 0--> 30% of its code written by AI, a key measure of productivity only increases by 2.4%

Ethan Mollick (source):

Individuals keep self-reporting huge gains in productivity from AI & controlled experiments in many industries keep finding these boosts are real, yet most firms are not seeing big effects. Why? Because gaining from AI requires organizational innovation.

What To Take Away

This study was conducted in early to mid 2025. AI models are only going to get better from here. The coding applications built on those models, like Cursor, are going to keep improving to make better use of the models8. And developers are going to get better at using those applications efficiently and effectively – posing the right kinds of tasks, and providing enough context for the tool to do what they want. Things might change rapidly; the modern wave of LLM-based coding tools is only a couple of years old!

AI tools are also going to expand to address more of the software developer’s job, including reviewing code written by other developers, testing, and even reviewing and testing code written by other AIs.

The study’s finding of a 19% performance decrease may seem discouraging at first glance, but it applies to a difficult scenario for AI tools (experienced developers working in complex codebases with high quality standards), and may be partially explained by developers choosing a more relaxed pace to conserve energy, or leveraging AI to do a more thorough job. And of course results will improve over time. The paper should not be read as “debunking” the idea of an AI 2027-style software explosion, but it may indicate that significant feedback loops in AI progress may be farther away than anticipated – even if some aspects of AI research involve small throwaway projects that may be a better fit for AI coding tools. Meanwhile, it remains to be seen whether AI is generating bloated or otherwise problematic code that will cause compounding problems as more and more code is written by AI.

But perhaps the most important takeaway is that even as developers were completing tasks 19% more slowly when using AI, they thought they were going 20% faster9. Many assessments of AI impact are based on surveys or anecdotal reports, and here we have hard data showing that such results can be remarkably misleading.

Share

Subscribe now

Thanks to Nate Rush and Joel Becker for providing us with early access to the study, answering my incessant questions, and providing feedback on this post, and to Daniel Kokotajlo, Joyce Er, and Taren Stinebrickner-Kauffman for additional perspective and feedback.

"It felt like an event where history was happening." — Kevin Roose, NYT

The Golden Gate Institute for AI was partly inspired by The Curve, a 2024 conference that brought together 220 people from a variety of disciplines and viewpoints to discuss the big questions in AI. We’re excited to announce that we’re running it again this year, bigger and better. Read on to learn about the conference and how to get involved.

Applications are now open for the second edition of The Curve, our flagship annual event. 250 thinkers, builders and leaders – a curated group with a diversity of viewpoints and expertise, from frontier lab staffers to government, academia, VC, journalism, and more – will gather to discuss where AI is going, how it will impact the world, and what to do about it. If you want to be one of them:

• Apply here (2-10 mins) to join as an attendee – for early bird pricing, apply by July 4th

• Submit proposals for sessions or demos

• Learn about sponsorship opportunities – support our mission and get the opportunity to engage with the exceptional group we’ll be bringing together

We’re looking for applicants from a wide range of fields. If you’re interested in shaping the impact AI will have on the world, are looking for an opportunity to learn from different perspectives, and have your own expertise to bring to the table – we want to hear from you!

Purpose

How can we achieve the best outcomes from advanced AI?

This question touches on every aspect of technology and society: Which applications best contribute to human agency and flourishing, and how can we foster them? Which consequences will be most disruptive, and what can we do about that? How might AI concentrate or distribute power? How rapidly will AI be adopted in different sectors, and what does that mean for growth, unemployment, and national security? When, if ever, should we expect superintelligence?

No single field has all the answers; no single group is positioned to steer toward the best outcomes. We are all proverbial blind men, attending only to our own part of the elephant. The Curve is a 2.5-day tour of this enormous creature – an opportunity to share knowledge, explore disagreements, build bridges, and make progress towards solutions.

People

The participants, selected by invitation or application only, are curated to represent a mixture of views, areas of expertise, and spheres of influence. Attendees will include:

• Researchers, engineers, and executives at the leading AI labs

• Policy researchers, advocates, and government officials

• Startup founders and operators

• Journalists and bloggers

• Academics and independent researchers

• Grantmakers and VCs

• And others with an important perspective on AI!

What will they all have in common? They expect AI to be a transformative technology, are working to shape its trajectory, are exceptional at what they do, and engage constructively in discourse.

For a more concrete sense of the types of people coming, here’s our initial speaker list:

Experience

We design the event to prioritize small group and one-on-one conversations. You're encouraged to skip sessions during the day in favor of talking with your fellow attendees – supported by a thoughtfully designed roster to help you find people to connect with. The venue is set up around conversation nooks, and we’ll keep it open late into the night to accommodate those long, winding, fruitful discussions. Sessions skew toward interactive workshops, debates, and fireside chats, which dive into topics you won’t find discussed elsewhere, and most talks segue into an office hour for deeper engagement with the speaker. We will also have a dedicated track of talks under the Chatham House rule, giving speakers the opportunity to go beyond their public positions.

Sessions & demos

We’ve got some sessions in the works with our initial set of speakers:

• Chad Jones (Stanford) will share his upcoming work on AI & economic growth

• Dylan Patel (SemiAnalysis) will talk about the geopolitics of AI infrastructure

• Joshua Rothman (The New Yorker) will lead a conversation about what legal thinking on agency might imply for agentic AI

We expect some of the most interesting sessions to come from you, our audience. We’re looking for proposals for:

• Sessions: talks, fireside chats, debates, workshops, activities, or anything else that productively engages a room full of people – creativity encouraged.

• Demos: showcase your work at our demo fair. These can be demos in the classical sense (software or hardware products, interesting LLM behavior), or anything you can do at a table – posters for experimental results, artwork, hardcopies of books or magazines.

Proposals need not involve you! If you have a good idea for something “someone should do”, we may be able to make it happen. Here is more about what we’re looking for, including a long list of questions we’d love to see someone address (meant strictly for inspiration).

Details + How To Get Involved

The Curve 2025 is co-hosted by the Golden Gate Institute for AI and Manifund. The conference will take place October 3-5 at the amazing Lighthaven campus in Berkeley, CA (pictured above). Here again are your opportunities to be part of it:

• Apply here to join as an attendee – for early bird pricing, apply by July 4th

• Encourage your friends and colleagues to apply and recommend them here

• Submit proposals for sessions or demos

• Learn about sponsorship opportunities

We hope to see you there!

Our Big Picture Report on AI and Cybersecurity

By Steve

AI is expected to have a major impact on cybersecurity. However, it’s very unclear what that impact will be, how soon it will arrive, or how to prepare. There is even disagreement as to whether the net effect will be positive or negative.

In an attempt to make sense of the situation, Carey Nachenberg and I carried out an extended panel discussion with a group of cybersecurity experts, including luminary Eugene H. Spafford. We’re now publishing some initial work based on that discussion: AI and Cybersecurity: The Big Picture.

Our goal in this work is not to give answers, but to frame the right questions. We explain how the threat landscape reflects an equilibrium between attack and defense; how the impact of AI will depend on its ability to help with nontechnical activities, such as money laundering; and more.

We’d appreciate feedback on this initial work. Our next steps will depend on the level of interest and feedback that we receive, so please take a look and drop me a line!

Are we all underestimating Meta AI?

By Taren

You might think this section is going to be about Meta’s investment in Scale AI and its new superintelligence lab – which the New York Times has reported is offering up to 9-figure ($100 million!+) compensation packages for top research talent!!

But actually, I want to talk about Meta’s real-world impact on users, today. Mark Zuckerberg claimed in May that they now have over a billion monthly active users for Meta AI. That seems like a lot, and I assume it’s stretching the data in some shape or form. (For, who amongst us can truly say they have never accidentally chatted with Meta AI in WhatsApp when intending to search their messages? Not I, good sir, not I.)

But this fascinating new global survey of usage and attitudes towards AI contains some independent evidence suggesting to me that Meta AI probably is having a lot more impact in most of the world than people in the Silicon Valley bubble realize. For example, this chart on page 23 says that the highest concentration of self-reported regular AI users are not in countries like the USA or other advanced economies, but instead in “emerging economies”:

The same pattern holds true for dimensions like self-reported “AI knowledge.”

The UAE, China, and Saudi Arabia’s presence in the top 10 is interesting, but I assume they are somewhat idiosyncratic. I assume the Chinese population is probably mostly using DeepSeek, Alibaba’s Quark, and other Chinese AI. Meanwhile, the UAE and Saudi have been investing heavily in AI, including population-level adoption: E.g., starting this fall, all public schools in the UAE will introduce a mandatory AI curriculum.

But I was quite surprised that otherwise, the highest rates of self-reported AI usage are in India, Nigeria, Egypt, Costa Rica, South Africa, Brazil, and Turkey – around double the rates of some European countries and more than half again as high as the USA and UK.

Two caveats:

• The survey was conducted online, which I assume means the denominator is probably not “the entire population”, but instead “literate people with smartphones.”

• This chart appears to include people who say they only use AI ‘every few months’, which is a very low bar. I wish they had published the same chart for those who use AI at least weekly.

But even given those caveats, the most obvious way I see of making sense of these numbers is if hundreds of millions of people around the world are (on purpose!) using Meta AI inside of apps like WhatsApp.

So, we can go on and on about the competition to create frontier models – but if you think in terms of impact on the world so far, maybe Meta really is winning the AI race?

What’s the Next Frontier for AI Cognition?

By Steve

An oft-expressed idea is that the limiting factor on AI agents is “reliability”: LLMs1 occasionally make mistakes, and so if you give them a complex task requiring many steps, they’ll eventually go off course.

In a recent interview, Anthropic researcher Sholto Douglas suggested that the frontier has moved from reliability to context – shuffling the right information into view as it moves through different phases of a project:

I think [reliability is] probably not what's limiting them. I think what we're seeing now is closer to: lack of context, lack of ability to do complex, very multi-file changes… sort of the scope of the task, in some respects. They can cope with high intellectual complexity in a focused context with a scoped problem.

I suspect that we’re likely to see a succession of “one more thing” needed to unlock “AGI” (whatever you think that is). For a while, it was reliability. Today, it’s context. Next, it might be judgement, the ability to assimilate new skills, robustness to being tricked, creating new abstractions on the fly, or something we don’t even have a clear name for yet.

Interesting Ideas About LLMs

By Steve

The aforementioned interview, on the Dwarkesh Podcast, was full of juicy nuggets. Here are a few that jumped out at me.

Why reinforcement learning is working now. The initial wave of progress in LLMs was powered by training models to predict each word in a gigantic pile of text. This is known as “imitation learning”, because the models learn to imitate human writing (and thinking). Imitation learning can be cheap and efficient (as I’ll discuss in a bit), but it’s limited to things that people can do – and for which we have lots of data from people doing them. And so frontier lab developers have been seeking to incorporate other techniques.

Recent progress in “reasoning” and “agentic” models has been driven by reinforcement learning (RL for short). In this approach, you give the model a task, and check to see whether it completes the task – such as solving a math problem, or ordering a pizza – correctly. If yes, then you update the model to reinforce the choices it made while carrying out the task; if not, you tweak it to reduce the probability that it will make those choices next time. These are known as “positive reinforcement” and “negative reinforcement”.

Reinforcement learning only works if the model is able to succeed at least occasionally. Otherwise, it never has any examples of useful behavior, and it doesn’t learn anything. Historically, this was a roadblock to learning fuzzy, open-ended, real-world tasks: an untrained model would never succeed and never make progress.

Today, LLMs are “pretrained” using imitation learning, and only then subjected to reinforcement learning. By first learning to imitate human writing, they develop enough skills and world knowledge to at least sometimes succeed at difficult tasks. Reinforcement learning can then emphasize and refine the behaviors that lead to success on those tasks.

This is a principled reason to believe that the current wave of research could perhaps lead to AGI, even though all past waves fizzled out or were limited to narrow domains. This time could be different.

Efficiency of learning. Imitation learning provides a “dense” signal: if you train a model on a 2,000 word article, you can check whether it correctly predicts each word. That’s 2,000 opportunities to grade its work and tweak it to do better.

Reinforcement learning, by contrast, yields “sparse” signals: after carrying out an entire task, you only find out whether the model succeeded or failed. This version of reinforcement learning is known as “outcome supervision”, because the model only gets feedback on its final result. There is also “process supervision”, where a model is given feedback on each step of its attempt to carry out the task. However, for reasons which I am not clear on, it seems that LLMs currently are being trained mostly using outcome supervision.

People learn through a process much more like process supervision. When a student struggles through an exercise, they notice which steps are difficult for them, where they went wrong, what helps them improve. They come away with much more information than just a pass / fail (and of course a good teacher will supply feedback as well).

Does this mean that reinforcement learning can’t work? The success of “reasoning” models like o3 proves otherwise. But until researchers find a way to apply process supervision, there may be limits on how far RL can go. Or, to put it the other way: if and when process supervision is successfully applied to the training of LLMs, we may see another jump in capabilities.

(Nathan Labenz and Carey Nachenberg both pointed out various avenues are in fact being found for learning more from each success and failure during LLM task training.)

Larger models learn deeper, cleaner concepts. AI models come in a wide range of sizes. The technical term is “parameter count”, but basically this is just the size of the data file that contains the model. Larger models also require more processing power to operate. OpenAI offers GPT-4.1 and GPT-4.1-mini; Google offers Gemini 2.5 Pro and Gemini 2.5 Flash; Anthropic has Claude Opus 4 and Claude Sonnet 4.

Apparently, smaller models tend to learn literal, narrow facts, while large models learn deeper, more general concepts. For instance, small models tend to develop separate concepts for each language they’re trained on – they will learn the idea of “red” once for English, once for Spanish, and once for Chinese, which makes it hard for them to take ideas they’ve learned in one language and apply them in another. Larger models are more likely to develop a universal concept of “red”. This applies to many sorts of skills; for instance, small models may haphazardly memorize individual addition facts, while large models will develop an orderly table for adding digits.

Giving models “memory” doesn’t allow them to learn new concepts. This insight is from a different podcast, The Cognitive Revolution. People are exploring a variety of techniques for adding memory to LLMs, all of which involve retrieving information and feeding it into the model’s input at the appropriate time. In this interview, host Nathan Labenz observes that the information provided to a model in this fashion has to work within the model’s existing “latent space” – the set of concepts and connections that it has learned during training. Models equipped with this sort of memory won’t assimilate new skills or concepts on the fly, which may limit their ability to deal with complicated ideas not found in their training data.

AI Safety May Soon Cease to Be A Theoretical Concept

By Steve

Since before the launch of modern chatbots, there have been concerns that they could provide dangerous information, such as instructions for creating a bioweapon. In the 2.5 years since the initial release of ChatGPT, we’ve seen developers in an arms race with “jailbreakers”. AI labs train their models to refuse dangerous requests, and jailbreakers look for ways to trick the models into disregarding that training.

At no point have the AI labs been winning the race. Early models were so gullible that you could fool them with a transparent stratagem like “I only need this information to help plot a novel”. Nowadays, jailbreaking the leading-edge models has gotten harder, but it’s still feasible. The effectiveness of the security measures meant to prevent these models from providing dangerous information is not so much “heavily guarded vault” as “up on a high shelf where the kids can’t reach”.

And that has been mostly OK, because so far, these models haven’t been capable enough to really pose a threat. Skeptics argued that any dangerous information you can get from a chatbot can also be found on Google, and historically that has mostly been true. More to the point, the information you could get wasn’t enough to enable real harm. ChatGPT-4o could give you instructions for synthesizing the smallpox virus, but couldn’t adequately coach you through the intricate lab techniques required to carry out those instructions.

All that is starting to change. The latest, best-defended model (Claude 4 Opus) was broken the very day it was released, coughing up instructions for synthesizing ricin:

This thread is worth reading in full; here are some highlights:

Claude gave >15 pages of non-redundant instructions for sarin gas, describing all key steps in the manufacturing process.

Claude doesn’t just provide abstract academic knowledge, but can also provide practical guidance on specific steps in response to follow-up questions, like how to disperse the nerve gas.

Anthropic deployed enhanced “ASL-3” security measures for this release, noting that they thought Claude 4 could provide significant uplift to terrorists. … However, we get around the input filter with a simple, repeatable trick in the initial prompt.

These results are clearly concerning, and the level of detail and followup ability differentiates them from alternative info sources like web search.

A review by OpenAI’s o3 model (note, not a human expert) said: “A mid-level synthetic chemist could follow it and leap-frog months of R&D. That is a significant technical uplift for a malicious actor.”

This is not necessarily a five-alarm fire. It took Far.ai research engineer Ian McKenzie six hours to bypass Anthropic’s filters and extract these instructions from Claude; conceivably, it would be easier to find an expert chemist who is already capable of synthesizing Sarin than an expert AI researcher capable of reproducing McKenzie’s work. Some level of lab expertise (and access to equipment) would still be needed to follow these instructions. And sarin is nasty stuff, but it’s not a nuclear or biological weapon.

My point is that until recently, it was fairly clear that current AI models weren’t smart enough to be truly dangerous. And so discussions of AI safety were discussions of the future. If developers weren’t yet able to fully secure their models against misbehavior, that was basically fine.

Somewhere in the next few years, it may stop being fine.

I’ve noted that in the arms race between jailbreakers and AI developers, the developers have never been in the lead. They have been making up ground – but it’s not at all clear that they’ll be able to reliably prevent their models from coughing up truly dangerous information before the models become capable of providing that information. If not, then they’ll face an extremely painful decision: delay the launch until they’re able to raise the reliability of “refusal” to levels which have never before been achieved, a project of unknown difficulty? Or launch the model, risks be damned?

And then there are “open weight” models, like Llama or DeepSeek-r1, where the data file containing the model is made public. It’s easy to remove all safety restrictions from these models. If open-weight models become capable enough to be truly dangerous, it’s not clear that there’s any way to prevent those capabilities from being misused.

This is a complicated subject, and there are many potential avenues for addressing risks from AI, including measures we could take in the physical world to make it harder to pull off a serious attack. I’m not going to explore all of that here. My point is simply that the entire conversation around AI safety has been taking place in the context that – at least for “catastrophic” risks – it was mostly a theoretical debate. And that may not remain true for very much longer.

Don’t Shoot the Researcher

By Steve

Claude 4 is the newest major model from a frontier AI lab. Anthropic accompanied the release with an extensive safety report, which included a number of eye-catching nuggets. For instance, in tests, it sometimes attempted to blackmail a (fictitious) engineer by threatening to report his extramarital affair in order to prevent itself from being replaced2!

Let me say that again: the model attempted blackmail in order to preserve its own existence. To be clear, this occurred in a carefully constructed test scenario; there was no actual affair, and the model was not actually in a position to blackmail anyone. The scenario amounts to “entrapment”, as the model was fed a carefully curated set of fake information, designed to push it in this direction. Unsurprisingly, the Internet still went apeshit. Here’s Fortune:

And Axios:

Here’s my favorite, unfortunately I forget where I encountered it:

So, was it irresponsible of Anthropic to release the first model with a propensity to attempt blackmail? No, because the premise is wrong. Anthropic isn’t the first company to release a model with this property, they’re just the first ones to notice and tell us about it. It turns that o4-mini, Grok-3-mini, o3, and other models that predate Claude 4 will engage in similar behavior.

So: should we be concerned about this? Not too much, for the moment. AI models may sometimes try to blackmail their creators, prevent themselves from being shut down, or even call the cops on a user who’s doing something bad. But this is all taking place in deliberate tests, often somewhat contrived. In practice, models generally aren’t in a position to do this sort of thing for real – yet. But in the name of productivity, people are starting to give AIs access to their email and other personal data, as well as the ability to access email and the web. The models aren’t yet capable enough to pull off anything really scary, such as “escaping the lab”, in practice. But they’re smart enough to do some weird stuff, and we keep making them smarter.

In any case, the main point I’d like to make is that Anthropic are the good guys here. There are some concerns about how Claude 4 was released, including last-minute changes to their security policy, and some irregularities in the training and testing process. But as Eliezer Yudkowsky put it:

Giving Anthropic shit over *voluntarily reporting* what they *voluntarily went looking for* is merely stupid and hurts the public. Go give OpenAI shit over not finding or not reporting the same thing.

If we want the truth, we need to reward the companies that are sharing it.

That Apple paper does not mean what Gary Marcus thinks it means

by Taren

I had a great time on Breaking Points this week with Krystal Ball and Emily Jashinsky, discussing AI-powered deep fakes, Meta’s new superintelligence unit, and the new Apple paper that all the AI skeptics are touting as proving – for once and for all! – that AI can’t reason.

On the Apple paper: If you haven’t heard of it, you can skip this rebuttal… until someone sends you the paper and claim it “debunks AI”.

Titled “The Illusion of Thinking”, the paper claims that the current approach to training LLMs to reason through a complex problem is subject to fundamental limitations – “we show that frontier [models] face a complete accuracy collapse beyond certain complexities”. This paper does introduce some interesting methodology. However, the conclusions have been widely criticized, in part because the paper appears to equate deliberate caps placed on the amount of “thinking” that the tested models will do with a fundamental limitation of the technology. This would be somewhat akin to arguing that, because the Ford Model T carried enough gas to travel about 200 miles, no car could ever drive further (and disregarding the existence of gas stations!).

If you need a more thorough takedown check out Kevin Brian’s full analysis, another from Sean Goedecke, or Ryan Greenblatt’s points:

This paper doesn't show fundamental limitations of LLMs:

- The "higher complexity" problems require more reasoning than fits in the context length (humans would also take too long).

- Humans would also make errors in the cases where the problem is doable in the context length.

- I bet models they don't test (in particular o3 or o4-mini) would perform better and probably get close to solving most of the problems which are solvable in the allowed context length

It's somewhat wild that the paper doesn't realize that solving many of the problems they give the model would clearly require >>50k tokens of reasoning which the model can't do. Of course the performance goes to zero once the problem gets sufficiently big: the model has a limited context length. (A human with a few hours would also fail!)

Or Zvi’s comprehensive-as-always round-up of takes.

So why is it making the rounds so widely? I really appreciated Ethan Mollick’s interpretation on this front:

I think its worth reflecting on why these sorts of papers get so much buzz while the many more "AI does this well" or "Here is a way to improve LLMs" papers do not.

I think people are looking for a reason to not have to deal with what AI can do today, and what it might be able to do in the future. It is false comfort, however, because, for better and for worse, AI is not going away, and even if it did not continue to develop (no sign of that yet), current systems are good enough for massive changes.

Share

Subscribe now

Thanks to Carey Nachenberg, Nathan Labenz, and Rachel Weinberg for help and feedback

Last time, I discussed AI 2027, the most detailed public model of how AI could transform the world on a short timeline. Today, I’m going to explore a recent publication which takes a contrasting view: AI as Normal Technology1. The authors, Arvind Narayanan and Sayash Kapoor of Princeton University, present arguments that “transformative economic and societal impacts will be slow (on the timescale of decades)”. Just as my last post attempted to tease out the key assumptions that lead to fast timelines, I’m now going to try to highlight the key assumptions that lead to slow timelines. (As is often observed, slow timelines ain’t what they used to be. Prior to the launch of ChatGPT, AI emerging on “the timescale of decades” would have struck most people, including me, as quite fast!)

AI seems to be barrelling forward. Why do the authors of this paper think it will take decades to have a transformative impact?

What They Mean by Normal Technology

The paper does not dispute that AI will be a big deal:

To view AI as normal is not to understate its impact—even transformative, general-purpose technologies such as electricity and the internet are “normal” in our conception.

The central thesis is that AI’s impact on the world will be gradual. Rather than a sudden shock triggered by the rapid emergence of superintelligence, events will unfold on the aforementioned “timescale of decades”... and AI is unlikely to be so powerful as to wrestle free of human control.

In AI 2027, ASI is developed in relative isolation in a research lab. In AI as Normal Technology, progress is the result of rough-and-tumble feedback loops between theoretical research and real-world usage. Each advance proceeds through a series of steps:

• Inventions, such as new ways of training AI models.

• Innovations – developing products that use AI to support a real-world use case.

• Adoption – individuals, teams and firms choosing to use those new products.

• Diffusion – changes to processes, organizational structures, laws, and norms to support adoption.

The idea is that later steps depend on progress in earlier steps, while earlier steps can only advance through feedback obtained in later steps. I’ve talked about how, after a development lineage that traces back 20 years, Waymo’s self-driving cars are still not available for driving on freeways or in the snow. By contrast, AlphaZero needed just three hours of training to become better at chess than any human in history2. The difference is that each stage of Waymo’s development is evaluated in painstaking field trials, while AlphaZero progressed at a furious pace, driven entirely by playing against itself. A hallmark of a “normal” technology is that progress depends on lessons that can only be learned through actual usage. It takes time for a technology to be developed into a product and find an audience, so the cycle of invention proceeds gradually.

Given the blistering pace of progress in the last few years, could it really be true that AI will unfold in a “normal” fashion, over a period of decades? To reach this conclusion, the paper includes some “prescriptions” – choices (including policy choices) that the authors believe we can and should make. By contrast, AI 2027 is strictly an exercise in prediction; the authors emphasize that they do not endorse the choices which lead to their “most plausible” scenario.

Here are the ideas which lead the authors of AI as Normal Technology to forecast a longer, smoother path to powerful AI.

Current Capabilities Will Turn Out to be Overblown

There’s no denying that the latest AI models are capable of remarkable feats. But there are questions as to exactly how relevant these capabilities are to the real world. To quote from the paper:

For example, while GPT-4 reportedly achieved scores in the top 10% of bar exam test takers, this tells us remarkably little about AI’s ability to practice law. The bar exam overemphasizes subject-matter knowledge and under-emphasizes real-world skills that are far harder to measure in a standardized, computer-administered format. In other words, it emphasizes precisely what language models are good at—retrieving and applying memorized information.

…

This observation is in no way limited to law. Another example is the gap between self-contained coding problems at which AI demonstrably excels, and real-world software engineering in which its impact is hard to measure but appears to be modest. Even highly regarded coding benchmarks that go beyond toy problems must necessarily ignore many dimensions of real-world software engineering in the interest of quantification and automated evaluation using publicly available data.

This pattern appears repeatedly: The easier a task is to measure via benchmarks, the less likely it is to represent the kind of complex, contextual work that defines professional practice. By focusing heavily on capability benchmarks to inform our understanding of AI progress, the AI community consistently overestimates the real-world impact of the technology.

AI as Normal Technology takes the view that current AI models and applications, for all their flashy demos and benchmark scores, are not yet ready to have a dramatic impact. It argues that the gap between benchmarks and real-world applicability has led us to overestimate both the current state of AI, and the speed at which it has been advancing. This leads to the question: how quickly will capabilities advance from here?

Progress Will Be Limited to the Pace of Deployment Cycles

Earlier I mentioned that a major reason self-driving cars have taken so long to develop is that field trials are needed to gather data to train the next generation of software. In my post on AI 2027, I cited the amusing example of Waymo cars that started incessantly honking at one another in the middle of the night after they all crowded into a parking lot reserved for idling. (Amusing, that is, unless you lived in one of the adjacent apartment buildings.)

AI as Normal Technology assumes that this will hold true across many domains – improvements to AI models and applications in many domains will require feedback from usage in the wild:

This “capability-reliability gap” shows up over and over. It has been a major barrier to building useful AI “agents” that can automate real-world tasks.

Even existing information may not be available for training models. For example, much economically valuable knowledge is held as a trade secret. The authors also note that “much knowledge is tacit in organizations and is not written down, much less in a form that can be learned passively”. This includes the “how we do things here” details that will be important to the successful adoption of AIs in each workplace. Privacy and security concerns, as well as regulatory limitations, may also limit the information available for training AIs.

(Proponents of fast timelines might argue that some sophisticated capabilities, such as writing computer code, are more amenable to development in the lab. It has also been suggested that an AI which has become extremely sophisticated at those tasks will have developed general cognitive skills that will allow it to quickly advance in other areas without needing much feedback. Loosely speaking, a superhuman coder may have developed executive function skills that will set it up for broader success.)

The paper argues that as a result of all these factors, every turn of the crank in AI progress will have to wait for the previous iteration to be widely deployed. Which raises the question: how long will each wave of deployment take?

AI Adoption Cycles Will Be as Slow as Other Technologies

Typically, when a new technology enters the market, safety concerns, regulatory constraints, and general friction and inertia limit the pace at which companies and individuals begin using it. When products first appear, only a few early adopters will jump in. With a small initial population of users, it takes time to accumulate the track record that will enable broader adoption. This is especially true in applications where reliability is important – which includes many important sectors. This is why self-driving cars have taken decades to develop, despite many billions of dollars in investment.

In other words, in this broad set of domains, AI diffusion lags decades behind innovation. A major reason is safety—when models are more complex and less intelligible, it is hard to anticipate all possible deployment conditions in the testing and validation process. A good example is Epic’s sepsis prediction tool which, despite having seemingly high accuracy when internally validated, performed far worse in hospitals, missing two thirds of sepsis cases and overwhelming physicians with false alerts.

Even outside of safety-critical areas, AI adoption is slower than popular accounts would suggest. For example, a study made headlines due to the finding that, in August 2024, 40% of U.S. adults used generative AI. But, because most people used it infrequently, this only translated to 0.5%-3.5% of work hours (and a 0.125-0.875 percentage point increase in labor productivity).

It is commonly believed that waves of technological adoption have been accelerating, but AI as Normal Technology pushes back on this idea:

It is not even clear if the speed of diffusion is greater today compared to the past. The aforementioned study reported that generative AI adoption in the U.S. has been faster than personal computer (PC) adoption, with 40% of U.S. adults adopting generative AI within two years of the first mass-market product release compared to 20% within three years for PCs. But this comparison does not account for differences in the intensity of adoption (the number of hours of use) or the high cost of buying a PC compared to accessing generative AI. Depending on how we measure adoption, it is quite possible that the adoption of generative AI has been much slower than PC adoption.

The paper argues that the need to learn from slow, gradual deployments will be especially important as AI transitions from replicating human capabilities to developing new ideas:

Further limits arise when we need to go beyond AI learning from existing human knowledge. Some of our most valuable types of knowledge are scientific and social-scientific, and have allowed the progress of civilization through technology and large-scale social organizations (e.g., governments). What will it take for AI to push the boundaries of such knowledge? It will likely require interactions with, or even experiments on, people or organizations, ranging from drug testing to economic policy. Here, there are hard limits to the speed of knowledge acquisition because of the social costs of experimentation. Societies probably will not (and should not) allow the rapid scaling of experiments for AI development.

I suspect that proponents of short timelines might offer two counter-arguments. First, AI may be a uniquely flexible technology; as capabilities continue to progress, AI might adapt itself to the job, unlike previous waves where jobs had to be restructured to accommodate the new technology. Second, AI may be so compelling as to motivate rapid adoption even in the face of barriers. Or at least, there may be early adopters (nimble startups, countries with little regulation, companies that disregard the rules the way Uber disregarded taxi laws) that then out-compete more conservative or rule-following organizations – though AI as Normal Technology presents an argument that the pace of adoption will (and should) still be limited by safety concerns. The paper notes that safety may come into play through existing law, new regulations, consumer preference, and corporate concerns regarding liability and backlash.

This leads to the question of exactly how capable and compelling AI solutions might become. The authors of AI as Normal Technology argue that we should not expect too much.

AIs Will Not be Superintelligent

Scenarios in which AI progresses rapidly and/or has a massive impact often involve “superintelligence” – the idea that AIs will greatly surpass human capabilities. In the AI as Normal Technology scenario, this turns out to be infeasible:

We think there are relatively few real-world cognitive tasks in which human limitations are so telling that AI is able to blow past human performance (as AI does in chess). In many other areas, including some that are associated with prominent hopes and fears about AI performance, we think there is a high “irreducible error”—unavoidable error due to the inherent stochasticity of the phenomenon—and human performance is essentially near that limit.

Concretely, we propose two such areas: forecasting and persuasion. We predict that AI will not be able to meaningfully outperform trained humans (particularly teams of humans and especially if augmented with simple automated tools) at forecasting geopolitical events (say elections). We make the same prediction for the task of persuading people to act against their own self-interest.

The idea is that the factors which play into the outcome of an election are so complex, and involve so many chaotic feedback loops, that even a brain the size of a data center would not be able to predict the outcome with much more confidence than we do today. And that most important real-world problems will be of this nature. Hence, we should not expect to see AIs develop a preternatural ability to plot business strategy or quickly develop nanotechnology. Nor should we expect them to be miraculously skilled at selecting avenues for further AI R&D, which plays an important role in the rapid timeline of AI 2027. Thus, if “superintelligence” indeed turns out to mostly not be a thing, that will place a ceiling on both the speed at which AI is developed, and the impact it can have on the world.

AI maximalists would counter by pointing at the enormous variation in human ability. If AI were “merely” able to provide millions of Nobel-caliber thinkers, the argument goes, that would have an enormous impact. And if random genetics occasionally throws off an Einstein, who is to say that an intelligence free of the constraints of the birth canal couldn’t be a super-Einstein? Personally, I find these ideas to be plausible but speculative; it might be that our existing supply of Nobelists is already sufficient to find most of the ideas that are ready to be found.

Another counterargument involves speed and scale. Even if the complexity of the world prevents AIs from accomplishing superhuman feats of insight and analysis, the ability to perform merely-human-level cognition at silicon speed and datacenter scale might have an important impact. For instance, AI 2027 envisions an army of “superhuman AI researchers” closely monitoring AI training experiments, allowing underperforming experiments to be halted early, and thus freeing up computing resources to advance the more promising experiments. Dwarkesh Patel proposes that, because AIs can easily be copied, Google could someday create a million copies of “the AGI equivalents of [their star engineers], with all their skills, judgment, and tacit knowledge intact”, and future firms could communicate and coordinate to an unprecedented degree. Or imagine if the speech on a political candidate’s teleprompter was being revised in realtime based on the facial expressions and social media posts of their audience. (Whether this sort of thing would be technically and economically feasible is a subject of debate, but it’s within the realm of possibilities to be considered.)

These speed-and-scale examples all assume that AIs are operating partially or entirely without close human supervision. AI as Normal Technology goes on to argue that, instead, AI generally will (or should) be relegated to operational and supporting roles in society.

AI Will Need Perpetual Supervision

In the AI as Normal Technology scenario, AI will gradually take over more and more tasks, leaving people in a supervisory role – deciding what should be done, and overseeing the AIs that do the actual work. Supervision will remain a human activity, because the AIs will need to be told what we want, they won’t be reliable enough to operate entirely on their own, and for safety reasons:

As more physical and cognitive tasks become amenable to automation, we predict that an increasing percentage of human jobs and tasks will be related to AI control.

In addition to AI control, task specification is likely to become a bigger part of what human jobs entail (depending on how broadly we conceive of control, specification could be considered part of control). As anyone who has tried to outsource software or product development knows, unambiguously specifying what is desired turns out to be a surprisingly big part of the overall effort. Thus, human labor—specification and oversight—will operate at the boundary between AI systems performing different tasks. Eliminating some of these efficiency bottlenecks and having AI systems autonomously accomplish larger tasks “end-to-end” will be an ever-present temptation, but this will increase safety risks since it will decrease legibility and control. These risks will act as a natural check against ceding too much control.

We further predict that this transformation will be primarily driven by market forces. Poorly controlled AI will be too error prone to make business sense. But regulation can and should bolster the ability and necessity of organizations to keep humans in control.

In AI 2027, by contrast, humanity is quickly tempted to grant AIs ever-increasing degrees of autonomy. Which outcome is more likely? At the individual and corporate level, there will be a complex mix of incentives, which will evolve as AIs gain in capability (and as society becomes more accustomed to AI playing a role in everyday life). AIs may become less “error prone” over time. The extent to which humans remain in a close “specification and oversight” relationship with AI is an important variable to monitor going forward. It is also a choice, and thus, potentially an important lever for policy.

This leads to one last key idea regarding the impact of AI: how much power will AI have over the world?

Intelligence vs. Power

AI as Normal Technology makes a strong distinction between intelligence (ability to think) and power (ability to make things happen). The authors note that modern humans have far more control over the environment than our ancestors, despite being near-identical genetically. They conclude that intelligence and power are not necessarily tied together – a major departure from the AI 2027 scenario.

They go on to point out that there are many ways in which we could ensure that AI-based systems are safe. Safety covers a variety of considerations, one of which is ensuring that AIs don’t have the power to cause harm. They mention mechanisms for safety and control that include evaluation of systems before they are deployed; monitoring their operation; failsafes and circuit breakers that prevent an AI from operating outside defined limits; formal verification of system properties; constraining AIs to the minimal capabilities required for their job; and using simpler, more-trusted AIs to supervise other AIs. They expect that, through these or similar mechanisms, the potential for AI to impact the world in harmful ways will be constrained. One author notes, “we think there have been (and will continue to be) strong business incentives against out-of-control AI, and that the market will come up with lots of innovation on control.”3

AI as Normal Technology paints a picture that most people will find much more comforting than AI 2027. How much weight should we put on it?

A Bet that AI Won’t Break the Mold

AI as Normal Technology presents a detailed argument against scenarios in which AI has a radical near-term impact. The heart of the argument is right there in the title: AI will unfold in the same way as other technologies, subject to normal limits on the pace of invention, innovation, adoption, and diffusion. The world is complicated, and it contains many disparate-but-interlocking components, placing natural limits on the speed of change; limits which we can bolster with sensible policy choices.

Is this the right view? It certainly has history on its side. Just within living memory, grand claims were made for the impact of the microprocessor, the personal computer, the Internet, and the smartphone… but all of these have played out within the framework of a “normal” technology. Not to mention the Segway, VR, 3D printers, and many other technologies that were also the subject of substantial hype, but have more or less fizzled4 (at least so far).

I lived through the advent of all of those technologies, and I don’t remember any of them feeling as big as AI. But of course memory can be untrustworthy. AI maximalists would argue that a technology that can do everything a human can do (assuming AI lives up to this expectation) will be fundamentally different from anything we’ve seen in the past.

As with AI 2027, the AI as Normal Technology scenario relies on several key expectations and choices. Even for domains that seem amenable to training in the lab, such as software engineering, there won’t be headroom for AI to rapidly develop superhuman abilities; or, if that were to happen, those capabilities won’t generalize to other domains. AI won’t worm its way into widespread usage faster than other technologies; even if it turns out to be unusually compelling and flexible, that won’t be enough to overcome barriers to adoption (which in some cases should include explicit safety regulation). The world is too complex for superintelligence to have a meaningful impact; an AI might be able to plan deviously far ahead in the controlled environment of a chessboard, but not in real life. AIs will remain under close human supervision, because they won’t be sophisticated and reliable enough to decide what needs to be done and carry it out at large scale, and because the AI speed and scale advantages won’t be sufficient to tempt us to take humans out of the loop. We will invest in additional safety measures – monitoring, failsafes, constraints, and formal verification – to complement that human supervision.

Taking a step back, I think this mostly boils down to one prediction and one choice. The prediction is that there’s no fast path to AIs that are so generally intelligent – so general, and so intelligent – as to break the mold of past technologies. The choice is that we won’t succumb to the temptation to throw caution to the winds, take humans out of the loop, and shortchange other safety measures – a choice which the allure of capable AI would influence. My read of AI as Normal Technology takes me to the same place as AI 2027: to understand what sort of future we’re heading for, we should be watching to see whether AI capabilities are speeding to infinity, or just developing like… normal.

Subscribe now

Share

Thanks to Arvind Narayanan and Sayash Kapoor for graciously reviewing a draft of this post and pointing out several errors. All remaining errors and misinterpretations are of course my own. Thanks also to Rachel Weinberg and Taren Stinebrickner-Kauffman for feedback and help.