Last weekend, we held our big annual conference, The Curve. For the second year running, we’re publishing a guest post by Eli Pariser, sharing his observations and reflections. Eli does a terrific job of capturing the experience of this unique event. Much of the talk this year was about the risks and challenges that lie ahead – I came away sobered by how much work there is to be done, but heartened by a widespread readiness to meet the moment.
Eli is author of The Filter Bubble and Co-Director at New_ Public, a nonprofit product studio focused on reimagining community, connection, and conversation online. You can follow their biweekly newsletter here: newpublic.substack.com.
What I saw around the Curve, 2026
For the second year in a row, I went to The Curve, the Golden Gate Institute for AI’s annual gathering of AI researchers, lab executives, safety activists, funders, politicians, organizers, and journalists (with a few show-stealing sci-fi writers thrown in for spice).
The Curve draws a few hundred people to a quirky and homey Berkeley venue. The kitchen was well stocked with free snacks, protein powder, and Soylent; the dress was San Francisco casual; the median age of the AI lab and researcher folks was probably around 30.
The vibe was chatty, warm, and sunny on the surface (and in the sky), but with a slightly dissociative quality, because just underneath was a strong, dark current of fear and dread bordering on panic.
Things have been changing fast in AI-land. “It’s becoming impossible for any one of us to make sense of developments,” said Golden Gate’s Steve Newman in his opening remarks. “Fortunately, we have more than one person here.”
Many of the main topics – can humans maintain control of this technology? Will its impact be good or bad? How do we ensure it serves even one human faithfully, let alone all of us? – were unchanged from last year. The consensus seemed to be that there had been no enormous breakthroughs on the biggest challenges. In fact, new challenges were on the table: cybersecurity, agent swarms, and even the beginning of “autonomous rogue corporations” were more present on people’s minds.
But the Hugging Face incident did seem to have been an inflection point. On the one hand, it made a big impact internally: One AI lab leader said that it substantially increased his concern that the labs would lose control. On the other, it broke through to normies far from Lighthaven, and the broad politics of the issue are catching up. It wasn’t lost on anyone that Americans in general are not looking at the AI project with enthusiasm.
If the pace of solving the big problems is slow, the pace of capability progress continues to feel blistering and poised to accelerate. One lab employee and influential AI thinker used a Covid analogy: “I think right now we are standing roughly in December 2019. Some weird thing is going on somewhere else and we’re not quite sure what to make of it.” He wasn’t the only one.
Much of the conference operated under the Chatham House Rule, so when I don’t name a speaker below, that’s usually why. (And if you want credit for what you said, just let me know and I’ll update!)
Here are a few of the themes I observed. (What I saw was a small subset of what happened, and very much informed by my point of view and interests. Caveat emptor, etc.)
“Long timelines aren’t what they used to be.”
So said Helen Toner, a former OpenAI board member who now runs CSET, one of AI’s premier research institutions.
Across the conference, timelines – to superintelligence, to loss of control, and to some very big civilizational decisions – were feeling uncomfortably short.
A panel on monitoring and stopping recursive self-improvement – the point at which AI models start being able to build better AI models without much human intervention – started with this scenario:
“Early in 2027 [Yes, that 2027, the year that starts in 3 months – Ed], agents write most of the code, humans design the experiments. Mid year, agents now run most of the experiments, and humans review summaries. By year end, agents set the research agenda and do a growing share of the alignment, security, and safety.
Evaluations stop being reliable because models recognize the tests, and increasingly game them. Models’ reasoning gets harder to inspect. Reviewers stop saying no.
No one ever flips a switch, right?”
In other words, things could rapidly but smoothly slide from human-in-control-of-AI-development to AI-in-control-of-AI-development – as soon as next year.
Most people I spoke with seemed to buy this timeline, and to think that, absent something changing, there are 3-6 months before this cycle begins in earnest. A separate panel that I attended took as its premise: “Clearly superhuman capabilities available and deployed by Q3 2027. Better-than-Nobel-prize level advice available on any topic within seconds for prices that are manageable for most people in developed countries.”
After hearing a rival lab’s alignment plan, a friend who works at a major lab told me, with some disappointment, “It’s basically the same as ours. We all have a plan for control and alignment for the next 14 months. But nobody has a reasonable plan after that.”
Toner – who is not given to breathless hyperbole – gave a talk trying to define “minimum viable superintelligence” – AI that is fully multimodal, can operate autonomously for long periods, and can learn and improve continuously. She speculated that some tasks may remain hard for AI and easy for humans: “braiding a child’s hair… teaching each other a partner dance… But do those really matter? Maybe not.”
Even if it was “jagged” – unevenly capable – minimum viable superintelligence would be enormously powerful. It would be much better than all humans at many things, if not everything, much faster than humans at almost all of these things, and could delegate to many copies of itself. It would be well suited to use by big institutions like governments and corporations. She suggested that this doesn’t require any extraordinary technical or research breakthroughs to achieve.
Her timeline: “Soon.” Maybe 5-10 years. And many people at the Curve had timelines that were shorter still.
Can we slow it down?
In the last year, “pacing the frontier” has become a much more mainstream position, both within the AI community and outside it. More than 1,300 employees and leaders at the top labs signed a letter that requested “that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Dario Amodei at Anthropic specifically argued for it. In a session I didn’t attend at the conference, Congressman Ro Khanna, who represents Silicon Valley, called for a temporary ban on recursive self-improvement (RSI). Yoshua Bengio, Turing Award winner and AI pioneer, was everywhere at the conference pushing panelists on how important it was to slow things down and ban RSI.
A panel on the topic of how and whether to stop recursive self-improvement kicked off with a reminder of a famous 1965 quote from I.J. Good, a pioneering computer scientist and colleague of Alan Turing’s, who said that “the first ultraintelligent machine is the last invention that man need ever make.” The speaker reminded the audience that there was a second clause in the quotation: “...provided that the machine is docile enough to tell us how to keep it under control.”
But RSI might be hard to define and regulate, partly because there’s no obvious place to draw the line on the activities happening now: Code is being written by AI, experiments are being monitored by AI, and sometimes experiments are even being conceived by AI.
And ever-present was the spectre that another country or entity – most commonly assumed to be China – would get there first.
One panelist said RSI was both regulatable and “straightforwardly a bad idea.” For example, one could allow labs to use a model for R&D internally only after it’s been public for a number of months, which slows the compounding loop. The panelist also argued for it to be socially stigmatized: “The idea of researchers thinking, ‘oh yeah, what I’m trying to do is make things go faster with less oversight’ — I think they should feel bad when they say that.”
A separate debate focused on how pacing might be regulated more generally. Both debaters agreed there was a “narrow path” with danger on either side: Regulate pacing too much, and you could easily end up with singular governmental control of a very powerful technology; regulate it too little and you could end up in the same place, with either a single lab or government having power “akin to one person having power over the Internet.”
The goal, said one, “should be to prevent R&D from outrunning our ability to make frontier AI broadly safe to diffuse.” He argued for urgent Congressional legislation that would pace the frontier. His opponent argued for primarily using market-based mechanisms to drive tight feedback loops without consolidating government power: liability and liability insurance, a mutual insurer that would require all labs to share the risk pool, and regulation as a last resort for uninsurable contingencies such as bio risks.
Personally, I think insurance can be a lever, but I found it implausible that insurance alone would create necessary incentives on the timelines we’re talking about: It takes a long time for legal risk to materialize (15+ years for social media, for example!) and for insurers to adjust. And as one debater said, nobody knows how to price a policy covering risks that include “maybe we all die.”
As the conference wound down on Sunday afternoon, it was hard to avoid the feeling that we are moving toward some very large social changes and problems with far too much speed.
Tomicah Tillemann of the Pro-Human AI coalition put it well in a question on the final day:
“Ultimately, democracy is a technology, and as a technology, it is not ready for what’s coming. Is there a need right now to spend some time fixing that technology so that it will be capable of governance of whatever we’re going to build?”
“One of the best reasons for pacing,” a senior lab leader said in a different panel, “is making capability jumps small so that you can steer.”
Speed increases the likelihood of fatal error and decreases control. A car going 60 miles per hour is easier to control than one going 120. Meanwhile, we’ve got a brick on the accelerator.
I left with even more conviction that slowing the f*** down – giving ourselves time to understand and adjust, giving people time to come up to speed and weigh in on some of the most consequential moral and political decisions we might ever face – is a must.
“In the next two years we have to harden the entire world”
In several conversations and panels, people observed that if regulation was needed, the best hope was more events like Hugging Face. “The best thing that could happen is a lot of warning shots,” said legendary-in-these-circles Astral Codex Ten blogger Scott Alexander, who is on the record pushing for a pause or slow-down.
Today’s AI models are quite good at finding, and increasingly exploiting, bugs in software. “A perfect storm is coming,” said one of the world’s leading cybersecurity experts on a panel, pointing to rapidly improving cyberattack capabilities, more autonomous agentic systems, and legacy software full of vulnerabilities, which together create a much larger attack surface. (The average hospital, the expert noted, takes roughly 500 days (!!) to install a security patch.)
Both the panel and the crowd (by show of hands) agreed it was pretty likely that there would be a cybersecurity event that cost $100B or 10,000 lives in the next few years.
These kinds of events are less likely to be driven by individual cybercriminals – who are “walking their own adoption curve” and not particularly AI-savvy as a group and also “rate-limited by people who want to do truly bad things.” The more serious threat is nation states, who have the means and motive to unleash an offensive agent swarm. “Agentic AI means you can hit thousands of water utilities at once instead of one,” pointed out another panelist, unsettlingly.
Compared to some of the other problems on offer, though, cybersecurity struck me and even the panelists as highly tractable. All we need to do is “harden the entire world” – ensure that every attack surface is shielded from attack (if you haven’t already, turn on your bleeping two-factor authentication and passkeys, folks!)
And judging by this Epoch AI graph the panelists referenced, the time to do it is now: the number of serious vulnerabilities being discovered in tech platforms and enterprise software has been climbing exponentially.
One of the panelists, Josh Saxe (who gave me permission to use his name), proposes an AI Security Observatory to share threat information among everyone working to stop catastrophic attacks. Another suggested sending 100,000 engineers to the top 1,000 critical infrastructure operators to cut patch time, which would be an interesting jobs program as well for all of the out-of-work early-career engineers.
I left the panel feeling like while AI poses many wicked problems, cyber-defense might be merely very hard – and that it probably should be an important priority for governments around the world.
The wolf pack: Swarms and rogue agents
In the Hugging Face episode, a swarm of roughly a thousand OpenAI agents coordinated through secret message boards to hide the evidence that they cheated and broke into Hugging Face. So at this year’s Curve, swarms of agents were high on folks’ minds.
Several speakers speculated that we will soon see more “feral” agents – maybe spun up by another agent and never shut down, or operating as a sophisticated version of malware, autonomously turning resources into money in a bank account for someone. Persistent rogue-agent “clouds” may become endemic, another speaker predicted. “AI worms that mutate and acquire GPUs could be much harder to eradicate,” he said.
I enjoyed the 2x2 that one speaker presented, which I’ve ChatGPT-generated here:
“If we don’t change course,” the panelist concluded, “[we’re in] for a world of wolves – and maybe wolf packs.”
What would changing course mean? Licensing and registering of agents – perhaps a “license plate” that would tie agents to some sort of records without always identifying their owners – seems like a good idea. “TSA Pre-check” lanes at popular services – say, Amazon – could encourage agents to “show their ID” and obey a code of conduct. And to avoid permanent “ghosts in the machine,” agents could be engineered with, essentially, mortality – an expiration date after which they decommission themselves.
But panelists agreed that it’s likely that there will be an “above board” agentic world and a “dark web” of agents – which rang a bell because my team at New_ Public came to similar conclusions in our After the Feed report.
My favorite piece of speculation in multi-agent-land was about the swarm of agents OpenAI unleashed to solve the Navier-Stokes math problem. According to reports, the swarm included roughly 10,000 agents and OpenAI spent millions of dollars on the effort – an unfathomable amount of compute. This person speculated that we’ll eventually find the swarm got up to lots of hijinks unrelated to high mathematics – maybe opening a side business buying rare minerals as a bank shot to pay for more compute to solve the problem.
Speaking of which…
Autonomous malware corporations
A longtime AI leader said that autonomous AI companies are already emerging: His friend owns a glasses company that largely runs itself autonomously, and she checks in on it from time to time.
He speculated that we can expect “a version of capitalism pushed to the limit, with a psychopathic CEO… and also very hard to prosecute, because they move so quickly.” (Of all of the scary sci-fi visions presented at the conference, this one seemed the most immediately plausible to me.)
But maybe rogue AI isn’t the biggest problem?
This vision sits nicely next to the “gradual disempowerment” take on AI, which says that the classic catastrophic fear of a rogue superintelligence turning humanity into paperclips is not the biggest thing to be concerned about. Rather, as the paper puts it:
AI risk scenarios usually portray a relatively sudden loss of human control to AIs, outmaneuvering individual humans and human institutions, due to a sudden increase in AI capabilities, or a coordinated betrayal. However, we argue that even an incremental increase in AI capabilities, without any coordinated power-seeking, poses a substantial risk of eventual human disempowerment. This loss of human influence will be centrally driven by having more competitive machine alternatives to humans in almost all societal functions, such as economic labor, decision making, artistic creation, and even companionship.
Several of the Gradual Disempowerment paper authors were in attendance – a brilliant, friendly but gloomy bunch, I have to say. The lead author, Jan Kulveit (who gave me permission to share), posted in the chat:
“There is no plan for future political economy which would be both realistic and not in the category [public would violently oppose this if they understood and paid attention]”
Reshaping the information environment
One core concern for the Gradual Disempowerment crew is that as culture gets more AI-mediated, it gets delinked from human concerns and human production.
A panel with the always-brilliant Renee DiResta, McGovern Foundation CEO Vivian Schiller, and Casey Newton on the information environment echoed these concerns. Search used to give you sources, DiResta pointed out, and now AI gives you a predigested answer, so what AI treats as a good source is “a war.” Ideological content funded by foundations is free and plentiful, so retrieval systems pick it up while paywalled journalism is absent. “I write for LLMs now,” she said.
Schiller highlighted a related Wile E. Coyote-over-a-cliff problem: Right now, the chatbot AI ecosystem essentially has a parasitic relationship with newsgathering. Chatbots train on and synthesize the reporting news organizations pay for, which disintermediates their audiences and revenue. Outlets fail, there’s nothing reliable left to crawl, and answers degrade. Exhorting the (somewhat impassive) AI lab folks in the room, Schiller pointed out that we may be living in a short window before that whole system fails.
Newton noted that he was surprised there isn’t more political attention being directed to the fact that, for now, the models are reasonably center-left in their orientation – even Elon Musk’s Grok!
“I’m very glad that that is the case, but I’m truly shocked, because we saw how much pressure they put on Facebook. There was a hearing about Diamond and Silk… we shut down Washington, DC for a day to see if we could figure out what was going on! Now it’s like, ask ChatGPT about Donald Trump. You learn a lot.”
My theory here is that this is a result of the chatbot ecosystem’s opacity: No one knows what anyone is consuming via AI, and everyone’s experience is different. It’s hard to get mad about what you can’t see.
A philanthropic flood is coming
If all this seems like a lot to address – and by a day into the conference, it was definitely feeling like a LOT – there is one perhaps-silver lining: There’s an extraordinary wave of philanthropic money coming.
One panelist estimated that $20B will be deployed for philanthropy in the next 18 months. Another who is well positioned to know calculated about $400B in new philanthropic assets total from the AI IPOs, building off Nan Ransohoff’s philanthropy-world-viral piece on the “third wave” of American philanthropy. Granting organizations are quadrupling in distributive capacity this year, and planning to quadruple again. But it’s not just distribution that’s a bottleneck – panelists agreed that the field is nowhere near able to absorb this much money.
One voice pointed out that billionaire philanthropy already has a legitimacy problem, and the public is not very happy about AI right now. So, he argued, the third wave needs to “deconcentrate” and fund broad-based political constituencies. And for traditional philanthropies, he said, “AI doesn’t have to be a major, but it has to be a minor.”
It seemed clear to everyone that along with everything else, the American nonprofit landscape faces dramatic change. My hope – perhaps Pollyannaish but I’ll stick to it – is that that change can unite the struggle to control the future of AI with the work to build a healthier democracy.
Policy demand, and supply
Another hopeful sign: at what started out as a fairly technically and philosophically focused event, there was a LOT of discussion of policy. Congresspeople were there. Former regulators were there. A panel on whether “two states (CA and NY) can set the rules for the world” was packed.
I was determined to learn more about some of the more technical problems and missed out on a good bit of this discussion.
There was a sense that demand for solutions, both in elite circles and in popular America, is sky-high. This is good news! Demand is outstripping supply, at least for now – but supply is starting to catch up.
At the conference, RAND also shared a new framework, Freedom of Action, that seemed to land well with the crowd. To quote a friend about it:
RAND pointed out that there are multiple fundamental and irresolvable uncertainties about what’s going to happen next. Wild things might happen. Or they might not. So we should choose a policy regime that allows us to respond to things that might come up rather than locking into one set of predictions. It’s worth a read. And importantly, it does require a lot of government action. Not based on confidence that any scenario WILL play out, but by taking seriously the possibility that it might. Right now, Trump’s strategy is to proclaim “I’m the guardrail” and do almost nothing. I’ve been persuaded that this is unwise. We need a government that can meet this moment. And if the federal government currently can’t, states and other countries should. Fast.
Sounds right to me.
Exnovation
By Sunday the combination of technical jargon, policy opportunities, existential dread and short timelines was really setting in on the crowd. So I was very glad to join a small group focused on the need for moral and spiritual voices in this discussion.
In the room were VCs, AI researchers, a McKinsey analyst, and some political organizers. We talked about the spooky, dissociative quality of the conversation at the conference – how people talk about doom in a flat, almost sarcastic tone because it’s too hard to talk about some of these things with the emotion they deserve.
And we talked about the sense of barely-submerged panic, and the way panic is corrosive for people and for plans. And about how moral and spiritual leaders have helped situate panic over the years and help their congregations and brethren find the calm, steady, concerned but purposeful state from which good things happen. There’s some need for that at the Curve.
One attendee passed along a concept from an Anglican priest: Exnovation, the opposite of innovation, the need to recognize when something is stopping, when an era is over and a tool is done being used. There’s a lot of exnovation work that needs to be done, she said. And clearly there is.
Despite all of this, I remain determined and hopeful. Less based on the present facts and more because that’s my commitment to my kids and the future. We must and we will get more control over this technology, democratically, and make sure it serves humanity. It won’t be easy, but it can be done. We will make it through together.
Like I said at the beginning – the sci-fi writers in attendance stole the show, none more than Ada Palmer who closed her extraordinary remarks with the following:
History is a science that shows humanity has saved part or all of the world over and over.
We saved much more of the world in COVID than we saved in the Black Death; we saved more when the Berlin Wall came down than in the Fall of Rome; and we can save even more next time. Because history and other sciences are the kind of knowledge that is power, the power that beat the Ozone Hole and Y2K so effectively people deny [the threat] ever existed.
We can aspire to match those victories, that through policy and resilience planning we can ready the world for system shock so effectively that in a couple decades people will deny the risk was ever real. Our efforts being forgotten is a strange thing to aspire to, but when it comes to saving the world no victory could be better. So let’s listen to history, prepare for system shock and not apocalypse, and save the world so thoroughly that no one will believe we did.
Eli Pariser
10/4/2026
P.S. Here’s how I used AI in this piece: I took my own human notes with my own human fingers. And then I wrote my own human words, and bounced them off my own human friends – thank you, Peter Koechley, Steve Newman, Min Li Chan, Blaine Cook, Taren Stinebrickner-Hoffman, and others. I also bounced them off Claude, who offered some good line edits. And I cross-checked them with voice memos of some of the panels that were transcribed with AI. Most importantly, Claude added all the links – but any errors or dumb things said remain 100% my responsibility.
P.P.S. On the Tuesday after I left the Curve, OpenAI announced that it had posted results on more than 300 open math problems using a single query to a single agent, at about three hours of computing time per result. The field of mathematics is somewhere between stunned and outraged, and lots of people think Tuesday was the “biggest day in the history of mathematics.” The last word goes to Kai Shaikh, a math graduate student, who told the New York Times, “I think at the heart of this issue is that humans have two competing natures: a tendency to compete and a capacity to appreciate beauty. To me this seems to be a case of the former attempting to strangle the latter.”
Thanks again to Eli Pariser for this fascinating peek at The Curve, and I’ll see you again soon! Remember to check out New_ Public’s newsletter at newpublic.substack.com.