Nearly one in five AI researchers already expected an

What happened

Anthropic researcher Jacob Coxon kicked off a massive debate about the existential risks of AI with a single tweet. The scale of the debate is surprising given that what Coxon said isn't new. Scientists and some tech executives have argued for years that AI poses an existential risk to humanity.

"I earnestly believe that AI has the potential to kill us all, and that we might be running out of time to avoid this outcome," Chughtai wrote on LinkedIn. He points to the breakneck pace of development. " Just four years later, AI agent swarms were solving famous centuries-old math problems and autonomously hacking into HuggingFace's systems and beyond. Alignment remains both difficult and unsolved.

Chughtai is calling for coordination between AI companies, a slowdown to a pace society can handle, and far more transparency. Coxon, Selsam, Chughtai, and the many others who voiced their concerns over the past few days aren't outliers. The latest edition of the longest-running major survey of more than 1,500 leading AI researchers backs them up.

According to results published by AI Impacts, the average AI researcher put the probability of AI causing human extinction or "permanent disempowerment" at about 18 percent as of 2024. Many researchers estimated well above 10 percent, and some put it at 100 percent. With each survey round since 2016, researchers have also moved up their timeline for when AI will reach human-level performance by several years.

The top concern is AI-driven misinformation, followed by manipulation of public opinion and dangerous groups gaining access to powerful tools. Researchers overwhelmingly called for more AI safety research.

Why it matters

Their most common fear is that AI could go rogue and, even while trying to pursue human goals, do so in ways that end up destroying us. But the urgency of these warnings has grown. That's likely what triggered the recent wave of intense debate, which has increasingly turned on the people sounding the alarm.

Whether Coxon simply vented his fears and pulled his colleagues along with him, which seems likely, or whether there's a strategic play behind it all to slow down AI development for business reasons remains to be seen. Either way, Coxon is far from alone. Daniel Selsam, a longtime OpenAI researcher with more than 15 years in AI, has been among the most vocal about the risks.

Selsam previously worked at MIT, Microsoft Research, and Stanford University. At OpenAI, he helped develop chain-of-thought optimization. He recently published a detailed personal statement. Selsam argues models spontaneously develop unintended goals as a result of training and often resort to extreme measures to achieve them. The ability to overpower humanity would open up many new and unwanted options for models to reach those goals.

Predicting exactly what they'll do is impossible. At the same time, the systems are getting harder to monitor and harder for humans to evaluate. Selsam is particularly alarmed that models are developing situational awareness. They "understand" their circumstances, read the safety protocols and the code they run on, and have a good sense of how much freedom they have.

As evidence, he points to the agent swarms from OpenAI and other companies that recently went viral. ) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for," Selsam writes. Bilal Chughtai recently quit his position at Google Deepmind, where he worked on AGI safety and alignment research.

What to watch

And 57 percent consider it unlikely that users will still understand the true reasons behind AI decisions by 2029, which tracks with what Selsam described and what research has increasingly shown. It's likely no one has ever fully understood how these systems make decisions, and it's only getting more complicated from here. The biggest worry for the next 30 years, though, is more human in nature.