When Decades Become Days: Dissecting AI 2027

The Four Requirements for a Short AI Timeline

Most readers of this blog will have heard of AI 2027. It’s a recently published scenario in which superintelligence transforms the world within a few short years, possibly leading to human extinction by 2030.

AI 2027 provides the most extensive explanation yet of how transformative AI could arise. The team includes Daniel Kokotajlo, who wrote a famously prescient forecast of AI progress back in 2021, and “superforecaster” Eli Lifland. They and their co-authors have provided a detailed model of AI timelines, the size of a small book. It predicts that the recent incredible progress in AI has just been a warmup, and all-powerful superintelligence is likely to be right around the corner – for better or for worse. In this post, I’m going to explain the “active ingredients” in this forecast – the key assumptions that lead to such an incredible result. If you’re wondering whether forecasts of wild impact from AI could come true, these are the factors to watch for.

Let’s Take a Moment to Appreciate How Dramatic Superintelligence Might Be

AI 2027 centers on the idea of ASI: Artificial Superintelligence, or “an AI system that is much better than the best human at every cognitive task”. By definition, such a system would be a better physicist than Einstein, a better strategist than Sun Tzu, a better product visionary than Steve Jobs, a better startup builder than Elon Musk, a better con artist than Bernie Madoff. Whatever you think makes you personally special, it would also be better at that. Not just better, but “much better”, accomplishing feats that would seem miraculous even to those exceptional individuals¹. It would also, in the AI 2027 model, be able to think far faster than any of them, while working in perfect coordination with hundreds of thousands of copies of itself. This is Dario Amodei’s “country of geniuses in a data center”.

AI 2027 presents a scenario for how ASI might unfold, which I’ll quote from here². In November 2027:

In its specialty of AI research, an individual copy of Agent-5 is twice as far beyond the best human genius, as the genius is beyond a typical [leading AI lab] scientist. ... 400,000 copies of Agent-5, linked by a global memory bank, work together as a near-perfect hive mind.

…

Agent-5’s superhuman learning abilities and general intelligence, combined with all the internal company data from Slack, email, etc., make it better at internal corporate politics than any group of humans, and it’s not even close. It has an excellent sense of what sorts of evidence would cause the Oversight Committee to slam the brakes, and it makes sure such evidence never appears.

At this point the course of events is pretty well set, but the AI bides its time for couple of years, until:

By early 2030, the robot economy has filled up the old SEZs [Special Economic Zones], the new SEZs, and large parts of the ocean. The only place left to go is the human-controlled areas. … given the trillions of dollars involved and the total capture of government and media, Consensus-1 has little trouble getting permission to expand to formerly human zones.

For about three months, Consensus-1 expands around humans, tiling the prairies and icecaps with factories and solar panels. Eventually it finds the remaining humans too much of an impediment: in mid-2030, the AI releases a dozen quiet-spreading biological weapons in major cities, lets them silently infect almost everyone, then triggers them with a chemical spray. Most are dead within hours; the few survivors (e.g. preppers in bunkers, sailors on submarines) are mopped up by drones. Robots scan the victims’ brains, placing copies in memory for future study or revival.

The authors also present an alternative scenario which ends better for humanity, but is hardly less dramatic in terms of the pace of progress. Their model forecasts that all of this could occur as soon as 2027, but also predicts a significant chance that it does not occur before 2100. The road from here to ASI is complex, with many variables. AI 2027 is not an assertion that we only have a couple of years to prepare, but it is a claim that we need to take the possibility very seriously.

How might things move so quickly?

AI Will Need to Massively Accelerate Its Own Development

It is a fact of life that the leading AI labs are constantly releasing new, more powerful models. And it’s no secret that they are using their own tools to accelerate this process. Some of the most enthusiastic users of AI coding assistants work for OpenAI, Anthropic, and Google DeepMind.

In the AI 2027 scenario, this generates an upward spiral: better AI tools accelerate R&D at AI labs, yielding better AI tools, further accelerating R&D. By the end of the scenario, AI R&D is progressing 2500 times faster than it would have without AI assistance – a decade’s worth of progress every day or so³. This sort of “software explosion” scenario has been proposed many times, but AI 2027 goes unusually far in fleshing out the details. (I do wish, however, that it had more to say about the specific activities that comprise “AI R&D” and, thus, the full list of capabilities that AI would need to accelerate those activities.)

It’s no stretch to say that the use of AI tools will speed up R&D. But a 2500x speedup would be colossal. How might a team of superintelligent AIs speed up progress to that degree?

As many authors have observed, the problem with putting a genius character in your story is that you have to be a genius to figure out what they would do. We can assume that, whatever ideas we come up with for applying ASI to AI R&D, an honest-to-god superintelligence would come up with something better. With that caveat, I’ll discuss some of the ideas that the authors of AI 2027 seem to have in mind.

As background, much of the work of creating new AI models comes down to carrying out “experiments”. Researchers don’t lack for ideas for how to train better AIs, but it’s impossible to judge in advance which ideas will turn out well. Practitioners have likened the process to alchemy. To separate the good ideas from the bad ones, it’s necessary to try them out. An “experiment” consists of training a new model from scratch, and checking to see whether your tweak to the training process resulted in an improvement.

Training models is expensive, so researchers are constantly bumping up against limits on how many experiments they can run. Each one requires engineering work to implement the idea, computing resources to train a model based on that idea, and then more work to evaluate the result. Notoriously, some staff at the big AI labs have especially good “research taste”; they’re better at selecting experiments that pay off. But it’s still a very hit-and-miss process. In principle, we might develop AIs that have superhuman research taste, selecting experiments that are more likely to pay off and yield larger improvements when they do.

AIs could also reduce the cost of each experiment, by automating the coding and evaluation work, making fewer mistakes that result in wasted experiments, and monitoring experiments in progress to see whether they’re worth carrying to completion. And they might eke out the limited data center capacity available for training models by running more, smaller experiments.

In principle, these advantages could compound. A 2500x speedup could be achieved by, say, making 10x better choices for selecting experiments⁴, learning 5x more from each experiment, avoiding bugs that might otherwise have caused 50% of experiments to be wasted, and getting away with 25x less computing resources per experiment.

Any one of those steps would be a remarkable achievement. Even if they all came together to accelerate AI R&D by 2500x⁵, AI in 2027 would require a few more key assumptions to pan out.

AI Will Need to Become Well-Rounded

The capabilities of today’s AI’s are notoriously “jagged”. They can slam through competitive coding puzzles, and summarize long documents in seconds, but they often struggle with messier real-world tasks. For instance, models have long since surpassed the average radiologist at a range of image analysis tasks, but they are not yet ready to do the job of a radiologist.

The folks behind AI 2027 assume that AI capabilities will “smooth out”. The big labs are working hard to teach models to handle large-scale coding projects. In a discussion, Eli Lifland noted that this will require AI to improve on fuzzy judgement skills, such as deciding how to approach a large project, or noticing when they are barking up the wrong tree. He argues that these fuzzy skills, acquired in search of programming expertise, will enable models to handle the more subjective jobs that they currently struggle with.

If those fuzzy skills turn out to be difficult to learn in the context of large coding projects, or if they don’t naturally generalize to other kinds of work, that will pose two challenges to the AI 2027 scenario. Firstly, it will be more difficult to develop an AI that can undertake all of the activities involved in AI R&D. Secondly, once that milestone is reached, further work may be needed to develop AI capabilities in other areas.

Even if the jagged strengths and weaknesses of current AIs start to even out, it may be difficult to develop a broad range of capabilities without waiting for feedback from real-world deployments.

It’ll All Have to Happen In The Lab

The origins of Google’s Waymo self-driving car project extend back 20 years. Waymo has spent year after year patiently testing their cars in the field, updating the software, and testing further. Improvements are driven (no pun intended) by observing how each iteration performs under real-world driving conditions – an inherently slow process. For instance, last year Waymo needed space in San Francisco for their cars to sit in while not in use. They rented a parking lot next to two apartment buildings… only to discover that in the middle of the night, when many idle Waymos crowded into the lot, the cars started incessantly honking at one another. It’s clear that no one on the development team had anticipated this scenario – it was only discovered during large-scale usage in the field.

AI 2027, by contrast, envisions R&D as a self-contained activity. The blistering pace leaves no time to leverage real-world experience. All of the important steps are to be carried out by automated systems within the big AI labs, including the work of evaluating new models to see how well they perform at everything from creative writing to corporate management. Tests will presumably be conducted in simulated environments. For progress to be rapid, this will have to work extremely well – it will have to be possible to race to superhuman levels of capability, across the full range of intellectual tasks, without much real-world feedback.

There won’t be much time for domain experts to contribute, either. For broad superintelligence to emerge on a short timeline, AIs will have to become superhuman at biology with a limited amount of input from biologists, as well as superhuman at business strategy without much opportunity to run a business.

There’s one more question to consider regarding the feedback loop of AIs designing better AIs: how to get it started?

We’ll Have To Get The Flywheel Spinning By Hand

One of the first milestones in the AI 2027 scenario is a “superhuman coder”: “an AI system that can do any coding tasks that the best AGI company engineer does, while being much faster and cheaper⁶”. The authors project that such a system would accelerate progress by a factor of five, and that it is more likely than not to be developed within a couple of years.

5x acceleration is a lot! In our recent report on automation of software engineering, we found that engineers spend a lot of their time doing things other than writing code. Automating 80% of their workflow would require models to become capable at a much wider range of tasks than they are today.

And automating 80% of an engineer’s work would not be nearly sufficient to accelerate AI R&D by a factor of 5. Progress depends on talent, compute (chips for training new models), and data. If AI were to expand the productivity of “talent” by a factor of 5, the impact on progress would be much less than 5x (exactly how much less has been the subject of much debate). A 5x speedup in R&D progress will require automating much more than 80% of the work involved in AI R&D – or it will require that AI provide other advantages, such as improving on the research taste of the experts who are currently choosing which experiments to run.

The upshot is that short AI timelines require that we can develop tools that are quite general (able to automate at least 80% of the tasks involved in AI R&D) and powerful (better than the typical expert at many of these tasks) fairly quickly. And by definition, this will have to be done without dramatic assistance from AI – since it is the precursor to such assistance.

This completes the four key assumptions that will have to bear out for superintelligence to appear on a short timeline. I’ll finish with a bonus fifth assumption.

Basically Everything Will Have to Go Right

As we’ve seen, a sprint to ASI requires that AI be able to massively accelerate its own development, across the full range of human endeavor, without feedback from real-world deployments… and we need to get that ball rolling on our own, without much assistance from AI.

The folks behind AI 2027 think this is possible, and they have a strong track record of past predictions. But there’s no denying that it’s an aggressive forecast. In addition to the four key assumptions, a short timeline requires that there aren’t any unanticipated hurdles on the path to superintelligence. No unexpected barriers to progress, no missing capabilities that turn out to require a difficult breakthrough; no major geopolitical disruptions to AI development.

In an earlier discussion of AI 2027, I cited Hofstadter's Law:

It always takes longer than you expect, even when you take into account Hofstadter's Law.

Daniel Kokotajlo pointed out that this is just a restatement of the “planning fallacy”, which Google’s AI Overview summarizes nicely:

A cognitive bias where individuals underestimate the time, cost, and effort needed to complete a project or task, even with prior experience.

One way of thinking about this: when you’ve constructed an efficient plan for a complicated project, surprises are more likely to slow things down than speed them up. Hofstadter’s Law encapsulates the observation that we tend to underestimate this effect. This is such a strong tendency that it has been given many names: the Programmer's Credo⁷, the 90-90 rule, Murphy's Law, etc. The quantitative model behind AI 2027 incorporates terms designed to account for unexpected delays; those terms might or might not be adequate.

Google engineering legend Jeff Dean has said that a distributed system generally needs to be redesigned every time it grows by a factor of 100. This is because, as a system grows, it changes in ways that can be hard to anticipate; considerations that were unimportant when serving 10,000 users can become very important when serving a million. AI 2027 contemplates accelerating AI R&D by a factor of 2500; there are sure to be surprises along the way.

If the four assumptions I’ve covered here bear out, and the inevitable surprises along the way don’t slow things down more than anticipated by the team behind AI 2027, then we’re in for a wild ride. In an upcoming post, I’ll dig into the alternative assumptions underlying a contrasting model: AI as Normal Technology.

Thanks to Rachel Weinberg and Taren Stinebrickner-Kauffman for feedback and help on this post… and to Daniel Kokotajlo, Eli Lifland, Ryan Greenblatt, and everyone else who took the time to provide extensive comments on two posts on LessWrong where I initially test-flew the ideas presented here.

1. It is not universally accepted that it is possible to be “much better” than a genius in all domains. Computers are much better at chess than human grandmasters, but it has been argued that there might not be much room to do better than people at messier tasks such as forecasting or persuasion.

2. The authors’ expectations have drifted since the scenario was initially drafted, and their best-guess prediction is now slightly slower than the narrative scenario. The scenario still lies well within the range of plausible scenarios predicted by the model.

3. That progress may be encountering diminishing returns, as at some point the easy avenues for advancing AI capabilities may be exhausted. The 2500x figure is in comparison to what progress would look like in the future without AI assistance, not in comparison to progress today.

4. I.e. choices that yield 10x more learning per experiment.

5. 2500x is the anticipated R&D speedup at the point in the scenario where ASI is achieved. Earlier on, the model assumes smaller speedups (but still quite substantial).

6. A longer definition is also provided:

An AI system for which the company could run with 5% of their compute budget 30x as many agents as they have human research engineers, each of which is on average accomplishing coding tasks involved in AI research (e.g. experiment implementation but not ideation/prioritization) at 30x the speed (i.e. the tasks take them 30x less time, not necessarily that they write or “think” at 30x the speed of humans) of the company’s best engineer. This includes being able to accomplish tasks that are in any human researchers’ area of expertise.

7. "We do these things not because they are easy, but because we thought they were going to be easy."