More than 1 in 10 chance AI ‘could kill all humans,’

What happened

A senior Anthropic safety researcher has said there is more than a 10 percent chance artificial intelligence “could kill all humans” by the end of the decade, just hours after a colleague resigned over fears the AI lab and its rivals are carelessly racing to build “superhuman systems” they cannot control.

Why it matters

In a post on X announcing his departure, Jacob Coxon, a researcher who has trained AI systems at Anthropic, said he had quit the company over its lax approach to safety. ” Industry insiders have long expressed concerns about the potential dangers of self-improving AI systems, which they warn could spiral out of human control in a runaway loop often described as recursive self-improvement.

While not yet realized, companies are actively pursuing this goal and much of today’s AI code is written with the help of AI. ” He also agreed with Coxon’s characterization. ” Despite this, Hubinger said Anthropic does “not yet have a plan” for ensuring advanced AI remains safe and aligned with human values and “are not clearly on track to” develop one either.

What to watch

” Coxon’s departure marks one of the most high-profile examples of an employee leaving Anthropic, a company founded by former OpenAI members following concerns over safety at the company. In recent years, multiple researchers have cited safety concerns as motivating their decision to leave OpenAI. Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.