Key takeaways

  • Kokatajlo points out that the drumbeat of concern was growing well before Coxon’s viral resignation, the numerous hacking incidents, and…
  • He attributes the recent flurry of concern to the specter of recursive self-improvement more than anything else.
  • That’s insane,’” Kokatajlo says.

What happened

Kokatajlo points out that the drumbeat of concern was growing well before Coxon’s viral resignation, the numerous hacking incidents, and the math breakthrough. Anthropic executives have said since the company’s founding that AI could represent an existential threat. In July over a thousand top AI engineers signed an open letter calling for a coordinated slowdown in the development of advanced AI.

One of the more easy-to-imagine scenarios could involve AI that is hooked up to a biolab, Soares suggests. ) AI hardly needs to wipe out humanity in order to be harmful, though. Many experts predict that more powerful models will lead to a coming wave of AI-assisted cyberattacks. The technology is now widely used for disinformation campaigns, and military adoption of AI is accelerating rapidly.

’ and it judges that, but we think that combining both AI and humans to do that task will lead to even better performance,” he says.

Why it matters

He attributes the recent flurry of concern to the specter of recursive self-improvement more than anything else. But it also comes at a time when people are concerned about massive data center build-outs and potential job losses from AI. Trust in AI companies—and AI researchers themselves—may be reaching an all-time low. “People are waking up and saying ‘the companies are actually trying to build superintelligence … what?

That’s insane,’” Kokatajlo says. Just how risky it is to carry on building AI is hard to quantify. But when pushed to explain exactly how AI might go about eliminating the species that created it, Soares suggests it could happen in a number of ways. It could involve manipulating humans to trigger a catastrophe, or controlling an army of killer robots.

What to watch

Still, not everyone sees doom as inevitable. Jain, the ex-Google DeepMind researcher, recently launched Sampura Research, a company working to develop techniques for aligning models that involve keeping humans in the loop, even if AI does the lion’s share of assessing whether behavior is good or bad. He notes that there is now significant funding for AI safety startups like his. Jain seems hopeful that AI can be tamed yet.