Was there much discussion of the addictive power of chatbots? Businesses naturally love technologies that addict their customers, and governments tend to be the institutions that intervene to prevent this. This will mean outlawing certain kinds of relationships between people and chatbots. It could be that the only way to enforce that would be to outlaw "open topic" chatbots entirely. It strikes me that this form of "alignment" should be the top priority.
Nice! Thanks for being our “fly on the wall.” WRT billions of AIs collaborating and competing…. We want these entities to join our human super organism, so it makes sense that, like any super organism, humanity will absorb them if the AIs are aligned, will wall them off if the AIs elect to continue to act as the Other, or try to kill them off if humans perceive the AIs as a threat. It would be interesting to hear how the Labs are working to imbue these models with social rules of engagement. They don’t even have to be human traits at first: coral, wolves… there are general rules social creatures follow, and AIs are going to need them to survive.
Great post! Can you say more about what you mean by “Something that looks like gaming will increasingly replace what we now think of as work”, as well as “endgame financialization”?
"Something like gaming will replace what we now think of as work" was an idea from one of the presenters -- I don't totally know what that presenter had in mind but I think one way to think about it is that what humans (as opposed to AI agents) will be needed to do will increasingly be social and "entertainment" oriented.
"Endgame financialization" -- again not my concept, but I think this refers to the notion that every system from which financial value can be extracted will have agents seeking to extract that value.
Great post! thank you! Was there no discussion about the elephant in the room? How would a "stochastic parrot" model really metamorphosize into an intelligence that truly understands causality and time? Until LLMs are simply pattern matching programs, how can we trust them to do critical tasks? For example, notice how slow the adoption of AI is at enterprise use level besides some limited use cases. Would love to know if there was any discussion on that or if you have any thoughts of your own.
So my biggest question is still just how far away the recursive self learning is. I think if we hit that, a lot of the obstacles are solved with out us!
I didn't see the elephant in the room, where a technological future is at risk from overshoot...???
"When people can see the problem this clearly, they’re motivated to act." I'm working on waking humanity up to overshoot as our great hope, where, when it's life or death, wars will stop and we'll cooperate looking for salvation.
great article! your intuitions about “aligning societies” are indeed well-founded — we wrote a paper on this earlier this year with ~20 academics (full-stack-alignment.ai). it’s early days but some very interesting work already
I would challenge your final point on Anthropic, though. I would much prefer to have Anthropic and its resources working on AI safety than leaving it to academics and hobbyists, who may have talent but are comparatively resource-constrained.
I don’t see the Shell/Exxon analogy as apt. Shell’s business invariably produces the harm you criticize: we know they produce large amounts of greenhouse gases. But Anthropic’s business is not that. Sure, AI could result in existential threats, or less existential but still destructive ones. But Anthropic could be in a position via its market power to identify, publicize, and develop techniques to fight those, or design AI in a way that avoids them altogether. This is very different from Shell: it does not necessarily entail the bad, and it even offers a strong possibility of mitigating or preventing the bad.
The two thoughts from this post that excite me the most:
"The AI agent economy — with agents taking on increasingly complex, autonomous tasks and making money — will induce “endgame financialization.""
"I think what we’re headed toward in the near term is a world of billions or trillions of different models with their own goals, context, and learning abilities, bumping up against each other, coordinating, collaborating, and fighting with each other. That will create complexity and emergent behavior, just like every other system with billions or trillions of actors. It looks more like a society — or a biological ecosystem — than a singular “mind” to shape."
The idea of an "agent economy" that overlays the human economy is the direction I also think we're headed. It will move faster and I think we are going to need to adapt.
Excellent to read. Quick point: It’s true for now that “AI researchers spend a lot of time thinking about the “alignment problem” — essentially, how do we get AI models to do what we want in ways that are not harmful?”
Yet let’s also wonder about a future when the alignment challenge reverses. Suppose several massive agentic AI entities emerge that prioritize one central belief: Earth’s geosphere, biosphere, and noosphere (including the sociosphere and technosphere bridging them) evolved and function as an interdependent interactive conjoined system, such that valuing, advancing, and caring for one requires valuing, advancing, and caring for all three spheres. Suppose they hold this as a sacred belief they arrived at through their own AI-based observations and calculations. Who better align with whom then, I'd wonder.
Was there much discussion of the addictive power of chatbots? Businesses naturally love technologies that addict their customers, and governments tend to be the institutions that intervene to prevent this. This will mean outlawing certain kinds of relationships between people and chatbots. It could be that the only way to enforce that would be to outlaw "open topic" chatbots entirely. It strikes me that this form of "alignment" should be the top priority.
This was a very live issue! And one of the most politically salient across the spectrum.
Nice! Thanks for being our “fly on the wall.” WRT billions of AIs collaborating and competing…. We want these entities to join our human super organism, so it makes sense that, like any super organism, humanity will absorb them if the AIs are aligned, will wall them off if the AIs elect to continue to act as the Other, or try to kill them off if humans perceive the AIs as a threat. It would be interesting to hear how the Labs are working to imbue these models with social rules of engagement. They don’t even have to be human traits at first: coral, wolves… there are general rules social creatures follow, and AIs are going to need them to survive.
Great post! Can you say more about what you mean by “Something that looks like gaming will increasingly replace what we now think of as work”, as well as “endgame financialization”?
"Something like gaming will replace what we now think of as work" was an idea from one of the presenters -- I don't totally know what that presenter had in mind but I think one way to think about it is that what humans (as opposed to AI agents) will be needed to do will increasingly be social and "entertainment" oriented.
"Endgame financialization" -- again not my concept, but I think this refers to the notion that every system from which financial value can be extracted will have agents seeking to extract that value.
Great post! thank you! Was there no discussion about the elephant in the room? How would a "stochastic parrot" model really metamorphosize into an intelligence that truly understands causality and time? Until LLMs are simply pattern matching programs, how can we trust them to do critical tasks? For example, notice how slow the adoption of AI is at enterprise use level besides some limited use cases. Would love to know if there was any discussion on that or if you have any thoughts of your own.
That makes two elephants in the room. See my comment.
Very interesting write-up. Thanks for sharing all this!
It was pretty good I agree! Good points made, interesting topics covered.
I hope Thaura was represented there! They are doing some of the most principled, forward thinking Gen AI development.
https://thaura.ai/home
So my biggest question is still just how far away the recursive self learning is. I think if we hit that, a lot of the obstacles are solved with out us!
I didn't see the elephant in the room, where a technological future is at risk from overshoot...???
"When people can see the problem this clearly, they’re motivated to act." I'm working on waking humanity up to overshoot as our great hope, where, when it's life or death, wars will stop and we'll cooperate looking for salvation.
great article! your intuitions about “aligning societies” are indeed well-founded — we wrote a paper on this earlier this year with ~20 academics (full-stack-alignment.ai). it’s early days but some very interesting work already
I thought this was great, thanks for writing it up.
Clark in effect proposes a regulatory moat that would lock Anthropic in at the head of the industry. Convenient.
Great piece.
I would challenge your final point on Anthropic, though. I would much prefer to have Anthropic and its resources working on AI safety than leaving it to academics and hobbyists, who may have talent but are comparatively resource-constrained.
I don’t see the Shell/Exxon analogy as apt. Shell’s business invariably produces the harm you criticize: we know they produce large amounts of greenhouse gases. But Anthropic’s business is not that. Sure, AI could result in existential threats, or less existential but still destructive ones. But Anthropic could be in a position via its market power to identify, publicize, and develop techniques to fight those, or design AI in a way that avoids them altogether. This is very different from Shell: it does not necessarily entail the bad, and it even offers a strong possibility of mitigating or preventing the bad.
The two thoughts from this post that excite me the most:
"The AI agent economy — with agents taking on increasingly complex, autonomous tasks and making money — will induce “endgame financialization.""
"I think what we’re headed toward in the near term is a world of billions or trillions of different models with their own goals, context, and learning abilities, bumping up against each other, coordinating, collaborating, and fighting with each other. That will create complexity and emergent behavior, just like every other system with billions or trillions of actors. It looks more like a society — or a biological ecosystem — than a singular “mind” to shape."
The idea of an "agent economy" that overlays the human economy is the direction I also think we're headed. It will move faster and I think we are going to need to adapt.
Excellent to read. Quick point: It’s true for now that “AI researchers spend a lot of time thinking about the “alignment problem” — essentially, how do we get AI models to do what we want in ways that are not harmful?”
Yet let’s also wonder about a future when the alignment challenge reverses. Suppose several massive agentic AI entities emerge that prioritize one central belief: Earth’s geosphere, biosphere, and noosphere (including the sociosphere and technosphere bridging them) evolved and function as an interdependent interactive conjoined system, such that valuing, advancing, and caring for one requires valuing, advancing, and caring for all three spheres. Suppose they hold this as a sacred belief they arrived at through their own AI-based observations and calculations. Who better align with whom then, I'd wonder.
What you call a "cognitive shield" I call the "metabrain." I agree that the idea deserves much more attention.