Anthropic CEO Amodei doubles down on open-weight risk

What happened

Anthropic CEO Dario Amodei is sticking to his view that capable open-weight models carry significant risks, but he's not against open weights in principle. " But Amodeiy says he has two "nightmare scenarios": authoritarian states, especially China, could build more powerful models than the US and use them to gain military superiority or surveillance capabilities. And powerful open models could be misused for cyber or bio attacks.

Critics counter that cutting off US chip sales to China mainly fuels the growth of China's own chip industry and that distillation is a legitimate and useful technique. Amodeli also wants mandatory safety tests for all powerful models, regardless of where they come from or how they're licensed.

On that last point, Amodei sees a growing consensus, pointing to recent proposals that would exempt less powerful models from startups and research. Amodei's comments follow reports that US authorities are weighing steps to make it harder for US companies to use Chinese open-weight models, along with a letter signed by many major tech companies opposing such moves.

Why it matters

AI labs know from their usage data that such attempts are already happening with closed models as well. Open weights are riskier, he argues, because they lack safeguards and a released model can't be recalled. While open models also empower defenders, Amodei sees a strong imbalance on biological threats, where sufficiently capable models could weaponize viruses quickly while building defenses would take years even in the best case.

Amodei is calling for stricter chip export controls so China can't train more powerful models than the US without US chips. He wants action against industrial distillation, which China reportedly uses to partly sidestep the chip restrictions and act as a "free rider" on US AI success.

What to watch

Critics accuse Anthropic of talking down open-weight models to protect its own business from cheaper competition. OpenAI is also said to still be lobbying against open weights, even though the company signed the letter.