Lab vs. Life: Dissecting “AI as Normal Technology”

What if AGI Can't Be Developed in an Ivory Tower?

Last time, I discussed AI 2027, the most detailed public model of how AI could transform the world on a short timeline. Today, I’m going to explore a recent publication which takes a contrasting view: AI as Normal Technology¹. The authors, Arvind Narayanan and Sayash Kapoor of Princeton University, present arguments that “transformative economic and societal impacts will be slow (on the timescale of decades)”. Just as my last post attempted to tease out the key assumptions that lead to fast timelines, I’m now going to try to highlight the key assumptions that lead to slow timelines. (As is often observed, slow timelines ain’t what they used to be. Prior to the launch of ChatGPT, AI emerging on “the timescale of decades” would have struck most people, including me, as quite fast!)

AI seems to be barrelling forward. Why do the authors of this paper think it will take decades to have a transformative impact?

What They Mean by Normal Technology

The paper does not dispute that AI will be a big deal:

To view AI as normal is not to understate its impact—even transformative, general-purpose technologies such as electricity and the internet are “normal” in our conception.

The central thesis is that AI’s impact on the world will be gradual. Rather than a sudden shock triggered by the rapid emergence of superintelligence, events will unfold on the aforementioned “timescale of decades”... and AI is unlikely to be so powerful as to wrestle free of human control.

In AI 2027, ASI is developed in relative isolation in a research lab. In AI as Normal Technology, progress is the result of rough-and-tumble feedback loops between theoretical research and real-world usage. Each advance proceeds through a series of steps:

• Inventions, such as new ways of training AI models.

• Innovations – developing products that use AI to support a real-world use case.

• Adoption – individuals, teams and firms choosing to use those new products.

• Diffusion – changes to processes, organizational structures, laws, and norms to support adoption.

The idea is that later steps depend on progress in earlier steps, while earlier steps can only advance through feedback obtained in later steps. I’ve talked about how, after a development lineage that traces back 20 years, Waymo’s self-driving cars are still not available for driving on freeways or in the snow. By contrast, AlphaZero needed just three hours of training to become better at chess than any human in history². The difference is that each stage of Waymo’s development is evaluated in painstaking field trials, while AlphaZero progressed at a furious pace, driven entirely by playing against itself. A hallmark of a “normal” technology is that progress depends on lessons that can only be learned through actual usage. It takes time for a technology to be developed into a product and find an audience, so the cycle of invention proceeds gradually.

Given the blistering pace of progress in the last few years, could it really be true that AI will unfold in a “normal” fashion, over a period of decades? To reach this conclusion, the paper includes some “prescriptions” – choices (including policy choices) that the authors believe we can and should make. By contrast, AI 2027 is strictly an exercise in prediction; the authors emphasize that they do not endorse the choices which lead to their “most plausible” scenario.

Here are the ideas which lead the authors of AI as Normal Technology to forecast a longer, smoother path to powerful AI.

Current Capabilities Will Turn Out to be Overblown

There’s no denying that the latest AI models are capable of remarkable feats. But there are questions as to exactly how relevant these capabilities are to the real world. To quote from the paper:

For example, while GPT-4 reportedly achieved scores in the top 10% of bar exam test takers, this tells us remarkably little about AI’s ability to practice law. The bar exam overemphasizes subject-matter knowledge and under-emphasizes real-world skills that are far harder to measure in a standardized, computer-administered format. In other words, it emphasizes precisely what language models are good at—retrieving and applying memorized information.

…

This observation is in no way limited to law. Another example is the gap between self-contained coding problems at which AI demonstrably excels, and real-world software engineering in which its impact is hard to measure but appears to be modest. Even highly regarded coding benchmarks that go beyond toy problems must necessarily ignore many dimensions of real-world software engineering in the interest of quantification and automated evaluation using publicly available data.

This pattern appears repeatedly: The easier a task is to measure via benchmarks, the less likely it is to represent the kind of complex, contextual work that defines professional practice. By focusing heavily on capability benchmarks to inform our understanding of AI progress, the AI community consistently overestimates the real-world impact of the technology.

AI as Normal Technology takes the view that current AI models and applications, for all their flashy demos and benchmark scores, are not yet ready to have a dramatic impact. It argues that the gap between benchmarks and real-world applicability has led us to overestimate both the current state of AI, and the speed at which it has been advancing. This leads to the question: how quickly will capabilities advance from here?

Progress Will Be Limited to the Pace of Deployment Cycles

Earlier I mentioned that a major reason self-driving cars have taken so long to develop is that field trials are needed to gather data to train the next generation of software. In my post on AI 2027, I cited the amusing example of Waymo cars that started incessantly honking at one another in the middle of the night after they all crowded into a parking lot reserved for idling. (Amusing, that is, unless you lived in one of the adjacent apartment buildings.)

AI as Normal Technology assumes that this will hold true across many domains – improvements to AI models and applications in many domains will require feedback from usage in the wild:

This “capability-reliability gap” shows up over and over. It has been a major barrier to building useful AI “agents” that can automate real-world tasks.

Even existing information may not be available for training models. For example, much economically valuable knowledge is held as a trade secret. The authors also note that “much knowledge is tacit in organizations and is not written down, much less in a form that can be learned passively”. This includes the “how we do things here” details that will be important to the successful adoption of AIs in each workplace. Privacy and security concerns, as well as regulatory limitations, may also limit the information available for training AIs.

(Proponents of fast timelines might argue that some sophisticated capabilities, such as writing computer code, are more amenable to development in the lab. It has also been suggested that an AI which has become extremely sophisticated at those tasks will have developed general cognitive skills that will allow it to quickly advance in other areas without needing much feedback. Loosely speaking, a superhuman coder may have developed executive function skills that will set it up for broader success.)

The paper argues that as a result of all these factors, every turn of the crank in AI progress will have to wait for the previous iteration to be widely deployed. Which raises the question: how long will each wave of deployment take?

AI Adoption Cycles Will Be as Slow as Other Technologies

Typically, when a new technology enters the market, safety concerns, regulatory constraints, and general friction and inertia limit the pace at which companies and individuals begin using it. When products first appear, only a few early adopters will jump in. With a small initial population of users, it takes time to accumulate the track record that will enable broader adoption. This is especially true in applications where reliability is important – which includes many important sectors. This is why self-driving cars have taken decades to develop, despite many billions of dollars in investment.

In other words, in this broad set of domains, AI diffusion lags decades behind innovation. A major reason is safety—when models are more complex and less intelligible, it is hard to anticipate all possible deployment conditions in the testing and validation process. A good example is Epic’s sepsis prediction tool which, despite having seemingly high accuracy when internally validated, performed far worse in hospitals, missing two thirds of sepsis cases and overwhelming physicians with false alerts.

Even outside of safety-critical areas, AI adoption is slower than popular accounts would suggest. For example, a study made headlines due to the finding that, in August 2024, 40% of U.S. adults used generative AI. But, because most people used it infrequently, this only translated to 0.5%-3.5% of work hours (and a 0.125-0.875 percentage point increase in labor productivity).

It is commonly believed that waves of technological adoption have been accelerating, but AI as Normal Technology pushes back on this idea:

It is not even clear if the speed of diffusion is greater today compared to the past. The aforementioned study reported that generative AI adoption in the U.S. has been faster than personal computer (PC) adoption, with 40% of U.S. adults adopting generative AI within two years of the first mass-market product release compared to 20% within three years for PCs. But this comparison does not account for differences in the intensity of adoption (the number of hours of use) or the high cost of buying a PC compared to accessing generative AI. Depending on how we measure adoption, it is quite possible that the adoption of generative AI has been much slower than PC adoption.

The paper argues that the need to learn from slow, gradual deployments will be especially important as AI transitions from replicating human capabilities to developing new ideas:

Further limits arise when we need to go beyond AI learning from existing human knowledge. Some of our most valuable types of knowledge are scientific and social-scientific, and have allowed the progress of civilization through technology and large-scale social organizations (e.g., governments). What will it take for AI to push the boundaries of such knowledge? It will likely require interactions with, or even experiments on, people or organizations, ranging from drug testing to economic policy. Here, there are hard limits to the speed of knowledge acquisition because of the social costs of experimentation. Societies probably will not (and should not) allow the rapid scaling of experiments for AI development.

I suspect that proponents of short timelines might offer two counter-arguments. First, AI may be a uniquely flexible technology; as capabilities continue to progress, AI might adapt itself to the job, unlike previous waves where jobs had to be restructured to accommodate the new technology. Second, AI may be so compelling as to motivate rapid adoption even in the face of barriers. Or at least, there may be early adopters (nimble startups, countries with little regulation, companies that disregard the rules the way Uber disregarded taxi laws) that then out-compete more conservative or rule-following organizations – though AI as Normal Technology presents an argument that the pace of adoption will (and should) still be limited by safety concerns. The paper notes that safety may come into play through existing law, new regulations, consumer preference, and corporate concerns regarding liability and backlash.

This leads to the question of exactly how capable and compelling AI solutions might become. The authors of AI as Normal Technology argue that we should not expect too much.

AIs Will Not be Superintelligent

Scenarios in which AI progresses rapidly and/or has a massive impact often involve “superintelligence” – the idea that AIs will greatly surpass human capabilities. In the AI as Normal Technology scenario, this turns out to be infeasible:

We think there are relatively few real-world cognitive tasks in which human limitations are so telling that AI is able to blow past human performance (as AI does in chess). In many other areas, including some that are associated with prominent hopes and fears about AI performance, we think there is a high “irreducible error”—unavoidable error due to the inherent stochasticity of the phenomenon—and human performance is essentially near that limit.

Concretely, we propose two such areas: forecasting and persuasion. We predict that AI will not be able to meaningfully outperform trained humans (particularly teams of humans and especially if augmented with simple automated tools) at forecasting geopolitical events (say elections). We make the same prediction for the task of persuading people to act against their own self-interest.

The idea is that the factors which play into the outcome of an election are so complex, and involve so many chaotic feedback loops, that even a brain the size of a data center would not be able to predict the outcome with much more confidence than we do today. And that most important real-world problems will be of this nature. Hence, we should not expect to see AIs develop a preternatural ability to plot business strategy or quickly develop nanotechnology. Nor should we expect them to be miraculously skilled at selecting avenues for further AI R&D, which plays an important role in the rapid timeline of AI 2027. Thus, if “superintelligence” indeed turns out to mostly not be a thing, that will place a ceiling on both the speed at which AI is developed, and the impact it can have on the world.

AI maximalists would counter by pointing at the enormous variation in human ability. If AI were “merely” able to provide millions of Nobel-caliber thinkers, the argument goes, that would have an enormous impact. And if random genetics occasionally throws off an Einstein, who is to say that an intelligence free of the constraints of the birth canal couldn’t be a super-Einstein? Personally, I find these ideas to be plausible but speculative; it might be that our existing supply of Nobelists is already sufficient to find most of the ideas that are ready to be found.

Another counterargument involves speed and scale. Even if the complexity of the world prevents AIs from accomplishing superhuman feats of insight and analysis, the ability to perform merely-human-level cognition at silicon speed and datacenter scale might have an important impact. For instance, AI 2027 envisions an army of “superhuman AI researchers” closely monitoring AI training experiments, allowing underperforming experiments to be halted early, and thus freeing up computing resources to advance the more promising experiments. Dwarkesh Patel proposes that, because AIs can easily be copied, Google could someday create a million copies of “the AGI equivalents of [their star engineers], with all their skills, judgment, and tacit knowledge intact”, and future firms could communicate and coordinate to an unprecedented degree. Or imagine if the speech on a political candidate’s teleprompter was being revised in realtime based on the facial expressions and social media posts of their audience. (Whether this sort of thing would be technically and economically feasible is a subject of debate, but it’s within the realm of possibilities to be considered.)

These speed-and-scale examples all assume that AIs are operating partially or entirely without close human supervision. AI as Normal Technology goes on to argue that, instead, AI generally will (or should) be relegated to operational and supporting roles in society.

AI Will Need Perpetual Supervision

In the AI as Normal Technology scenario, AI will gradually take over more and more tasks, leaving people in a supervisory role – deciding what should be done, and overseeing the AIs that do the actual work. Supervision will remain a human activity, because the AIs will need to be told what we want, they won’t be reliable enough to operate entirely on their own, and for safety reasons:

As more physical and cognitive tasks become amenable to automation, we predict that an increasing percentage of human jobs and tasks will be related to AI control.

In addition to AI control, task specification is likely to become a bigger part of what human jobs entail (depending on how broadly we conceive of control, specification could be considered part of control). As anyone who has tried to outsource software or product development knows, unambiguously specifying what is desired turns out to be a surprisingly big part of the overall effort. Thus, human labor—specification and oversight—will operate at the boundary between AI systems performing different tasks. Eliminating some of these efficiency bottlenecks and having AI systems autonomously accomplish larger tasks “end-to-end” will be an ever-present temptation, but this will increase safety risks since it will decrease legibility and control. These risks will act as a natural check against ceding too much control.

We further predict that this transformation will be primarily driven by market forces. Poorly controlled AI will be too error prone to make business sense. But regulation can and should bolster the ability and necessity of organizations to keep humans in control.

In AI 2027, by contrast, humanity is quickly tempted to grant AIs ever-increasing degrees of autonomy. Which outcome is more likely? At the individual and corporate level, there will be a complex mix of incentives, which will evolve as AIs gain in capability (and as society becomes more accustomed to AI playing a role in everyday life). AIs may become less “error prone” over time. The extent to which humans remain in a close “specification and oversight” relationship with AI is an important variable to monitor going forward. It is also a choice, and thus, potentially an important lever for policy.

This leads to one last key idea regarding the impact of AI: how much power will AI have over the world?

Intelligence vs. Power

AI as Normal Technology makes a strong distinction between intelligence (ability to think) and power (ability to make things happen). The authors note that modern humans have far more control over the environment than our ancestors, despite being near-identical genetically. They conclude that intelligence and power are not necessarily tied together – a major departure from the AI 2027 scenario.

They go on to point out that there are many ways in which we could ensure that AI-based systems are safe. Safety covers a variety of considerations, one of which is ensuring that AIs don’t have the power to cause harm. They mention mechanisms for safety and control that include evaluation of systems before they are deployed; monitoring their operation; failsafes and circuit breakers that prevent an AI from operating outside defined limits; formal verification of system properties; constraining AIs to the minimal capabilities required for their job; and using simpler, more-trusted AIs to supervise other AIs. They expect that, through these or similar mechanisms, the potential for AI to impact the world in harmful ways will be constrained. One author notes, “we think there have been (and will continue to be) strong business incentives against out-of-control AI, and that the market will come up with lots of innovation on control.”³

AI as Normal Technology paints a picture that most people will find much more comforting than AI 2027. How much weight should we put on it?

A Bet that AI Won’t Break the Mold

AI as Normal Technology presents a detailed argument against scenarios in which AI has a radical near-term impact. The heart of the argument is right there in the title: AI will unfold in the same way as other technologies, subject to normal limits on the pace of invention, innovation, adoption, and diffusion. The world is complicated, and it contains many disparate-but-interlocking components, placing natural limits on the speed of change; limits which we can bolster with sensible policy choices.

Is this the right view? It certainly has history on its side. Just within living memory, grand claims were made for the impact of the microprocessor, the personal computer, the Internet, and the smartphone… but all of these have played out within the framework of a “normal” technology. Not to mention the Segway, VR, 3D printers, and many other technologies that were also the subject of substantial hype, but have more or less fizzled⁴ (at least so far).

I lived through the advent of all of those technologies, and I don’t remember any of them feeling as big as AI. But of course memory can be untrustworthy. AI maximalists would argue that a technology that can do everything a human can do (assuming AI lives up to this expectation) will be fundamentally different from anything we’ve seen in the past.

As with AI 2027, the AI as Normal Technology scenario relies on several key expectations and choices. Even for domains that seem amenable to training in the lab, such as software engineering, there won’t be headroom for AI to rapidly develop superhuman abilities; or, if that were to happen, those capabilities won’t generalize to other domains. AI won’t worm its way into widespread usage faster than other technologies; even if it turns out to be unusually compelling and flexible, that won’t be enough to overcome barriers to adoption (which in some cases should include explicit safety regulation). The world is too complex for superintelligence to have a meaningful impact; an AI might be able to plan deviously far ahead in the controlled environment of a chessboard, but not in real life. AIs will remain under close human supervision, because they won’t be sophisticated and reliable enough to decide what needs to be done and carry it out at large scale, and because the AI speed and scale advantages won’t be sufficient to tempt us to take humans out of the loop. We will invest in additional safety measures – monitoring, failsafes, constraints, and formal verification – to complement that human supervision.

Taking a step back, I think this mostly boils down to one prediction and one choice. The prediction is that there’s no fast path to AIs that are so generally intelligent – so general, and so intelligent – as to break the mold of past technologies. The choice is that we won’t succumb to the temptation to throw caution to the winds, take humans out of the loop, and shortchange other safety measures – a choice which the allure of capable AI would influence. My read of AI as Normal Technology takes me to the same place as AI 2027: to understand what sort of future we’re heading for, we should be watching to see whether AI capabilities are speeding to infinity, or just developing like… normal.

Thanks to Arvind Narayanan and Sayash Kapoor for graciously reviewing a draft of this post and pointing out several errors. All remaining errors and misinterpretations are of course my own. Thanks also to Rachel Weinberg and Taren Stinebrickner-Kauffman for feedback and help.

1. As in my discussion of AI 2027, I’m not going to cover all of the ideas presented in AI as Normal Technology – only the discussion of the scale of AI’s impact and the timeline over which that impact emerges.

2. This comparison is found in the paper:

Consider self-driving cars: In many ways, the trajectory of their development is similar to AlphaZero’s self-play—improving the tech allowed them to drive in more realistic conditions, which enabled the collection of better and/or more realistic data, which in turn led to improvements in the tech, completing the feedback loop. But this process took over two decades instead of a few hours in the case of AlphaZero because safety considerations put a limit on the extent to which each iteration of this loop could be scaled up compared to the previous one.

3. Sayash Kapoor, private communication.

4. Yes, 3D printers are a useful technology, and the others have found their niches as well, but they’ve hardly been a revolution on the grand stage. I do hold out some hope for VR and 3D printers in the long run.