"In the spirit of total transparency and ethical uprightness, we shall keep all our comments anonymous." Nothing captures the hypocrisy of the AI safety movement in the big labs better than this. It also contributes to the very low perceived credibility of AI scientists and safety activists among the general public. The very idea that people who are calling for accountability are unwilling to be accountable for their own words is just mind-boggling.
I totally understand this sentiment. But I also think it was valuable to hear from them directly without the corporate PR people (or HR, for that matter) hovering over them.
In your post there is a picture that shows a timeline where a decent amount of people had timelines for RSI occurring in 6 months but the majority of people had timelines of 1 to 2 years.
In a recent post Daniel Kokotajlo said that he thinks that there is a 50% of the Superhuman AI Researcher milestone or RSI arriving by the end of 2028. He later clarified to me that while the Superhuman AI Researcher milestone or RSI *could* arrive in 6 months he thought that it was unlikely that said milestone would happen that quickly. Do you have a similar viewpoint? In other words when do you think that the Superhuman AI Researcher milestone or RSI will be reached?
With low confidence, I think it will be 5 to 10 years or more before crazy things start happening in the real world. Note that "crazy things in the real world" is likely a more distant milestone than "Superhuman AI Researcher" or "RSI".
Broadly speaking, my worldview is informed by AI as Normal Technology ideas around limits to diffusion speed, frictions on capability advancement in the long tail of domains, etc. For example, I could imagine that in 2027 or 2028 we see AI systems able to autonomously advance their own development, but only within a limited paradigm that results in further capability and efficiency advances on things like theorem proving and software engineering but does relatively little to advance taste, strategic judgement, sample-efficient learning, the ability to crystallize new concepts, and other areas where "jagged" capabilities lag today. In other words, the first round of RSI might turn out to be AI getting good at making AI good at things AI is already good at.
Just to clarify! You seem to think that RSI or the Superhuman Coder Milestone has laid out by the AI Futures Team will happen eventually, but your skeptical that it will happen in 6 months. Is this correct?
More precisely: I'm skeptical that full RSI – AI advancing its own capabilities without human assistance and without near-term bound – will happen anywhere near that quickly. Of course weaker versions of RSI are already here – AIs are able to make various improvements unassisted, and it even seems possible that (with some human-aided setup work) Astra 6.1 could create an Astra 6.2 that was an improvement, but I'd expect that chain to peter out very quickly (or go completely off the rails, creating benchmaxxxed models that really aren't any good) absent human assistance. More to the point, I don't expect a proper intelligence explosion to kick off soon.
I forget exactly how "superhuman coder" is defined by the AI Futures team so I'm not sure about when I'd expect it to be reached, but even if it is defined in a way that will be reached soon I don't expect that to have profound implications, per the above.
I - genuinely - wonder how many of the participants at the conference have read the book Messy Jobs (or something equivalent, AI as Normal Tech is close).
I am very interested in the cluster in the bottom right that said AI industry bubble would *never* cause a 20% market crash. The telecom crash of 2001 saw ~40% drops so that cluster is not people saying there will never be a crash _that large_, but that there will never be a crash at all.
I.e. these crowd has about 50% of people that truly think It's Different This Time.
I consider that an interesting proxy for people who are not empiricists, and ultimately unwise (if very intelligent) at this event. Same epistemic mistakes as Leopold over at Situational Awareness.
"In the spirit of total transparency and ethical uprightness, we shall keep all our comments anonymous." Nothing captures the hypocrisy of the AI safety movement in the big labs better than this. It also contributes to the very low perceived credibility of AI scientists and safety activists among the general public. The very idea that people who are calling for accountability are unwilling to be accountable for their own words is just mind-boggling.
I totally understand this sentiment. But I also think it was valuable to hear from them directly without the corporate PR people (or HR, for that matter) hovering over them.
Hi Steve!
I briefly scanned your post.
In your post there is a picture that shows a timeline where a decent amount of people had timelines for RSI occurring in 6 months but the majority of people had timelines of 1 to 2 years.
In a recent post Daniel Kokotajlo said that he thinks that there is a 50% of the Superhuman AI Researcher milestone or RSI arriving by the end of 2028. He later clarified to me that while the Superhuman AI Researcher milestone or RSI *could* arrive in 6 months he thought that it was unlikely that said milestone would happen that quickly. Do you have a similar viewpoint? In other words when do you think that the Superhuman AI Researcher milestone or RSI will be reached?
With low confidence, I think it will be 5 to 10 years or more before crazy things start happening in the real world. Note that "crazy things in the real world" is likely a more distant milestone than "Superhuman AI Researcher" or "RSI".
Broadly speaking, my worldview is informed by AI as Normal Technology ideas around limits to diffusion speed, frictions on capability advancement in the long tail of domains, etc. For example, I could imagine that in 2027 or 2028 we see AI systems able to autonomously advance their own development, but only within a limited paradigm that results in further capability and efficiency advances on things like theorem proving and software engineering but does relatively little to advance taste, strategic judgement, sample-efficient learning, the ability to crystallize new concepts, and other areas where "jagged" capabilities lag today. In other words, the first round of RSI might turn out to be AI getting good at making AI good at things AI is already good at.
Just to clarify! You seem to think that RSI or the Superhuman Coder Milestone has laid out by the AI Futures Team will happen eventually, but your skeptical that it will happen in 6 months. Is this correct?
Correct.
More precisely: I'm skeptical that full RSI – AI advancing its own capabilities without human assistance and without near-term bound – will happen anywhere near that quickly. Of course weaker versions of RSI are already here – AIs are able to make various improvements unassisted, and it even seems possible that (with some human-aided setup work) Astra 6.1 could create an Astra 6.2 that was an improvement, but I'd expect that chain to peter out very quickly (or go completely off the rails, creating benchmaxxxed models that really aren't any good) absent human assistance. More to the point, I don't expect a proper intelligence explosion to kick off soon.
I forget exactly how "superhuman coder" is defined by the AI Futures team so I'm not sure about when I'd expect it to be reached, but even if it is defined in a way that will be reached soon I don't expect that to have profound implications, per the above.
I - genuinely - wonder how many of the participants at the conference have read the book Messy Jobs (or something equivalent, AI as Normal Tech is close).
My very uninformed guess: 4% on that book in particular. But probably 50% are familiar with AI as Normal Tech.
I am very interested in the cluster in the bottom right that said AI industry bubble would *never* cause a 20% market crash. The telecom crash of 2001 saw ~40% drops so that cluster is not people saying there will never be a crash _that large_, but that there will never be a crash at all.
I.e. these crowd has about 50% of people that truly think It's Different This Time.
I consider that an interesting proxy for people who are not empiricists, and ultimately unwise (if very intelligent) at this event. Same epistemic mistakes as Leopold over at Situational Awareness.