Anthropic's fight with Pentagon shows how AI could

What happened

In February, the United States Department of Defence threatened to designate the AI firm Anthropic a "supply-chain risk" after a dispute over the military's use of the company's Claude models. At the centre of the dispute were two uses that Anthropic said it would not permit: fully autonomous weapons and the mass surveillance of Americans.

" Anthropic CEO Dario Amodei said the company did not believe current frontier AI models were reliable enough for fully autonomous weapons and argued that mass domestic surveillance would endanger democratic values. Anthropic challenged the designation in court, and on March 26, a federal judge issued a preliminary injunction blocking its enforcement while the litigation proceeds. The dispute raises a question larger than the future of one government contract.

I have previously argued that this includes a duty to resist autocratic demands. Here, I want to focus on a particularly urgent version of that responsibility: the duty to warn. Two kinds of warning Companies often know things that democratic citizens need to know in order to govern themselves. For example, Facebook's internal research raised serious concerns about Instagram's effects on teenagers' mental health.

Those findings became public only after former Facebook employee Frances Haugen came forward as a whistleblower. At one level, Anthropic is doing something similar. It is warning democratic society that it faces important choices about how governments use AI. Disinformation on social media raises a similar problem. It is not simply another problem for democratic society to manage.

It can attack the shared epistemic and social commitments that make democratic self-government possible in the first place. AI and the threat to democracy Much of the concern about AI and democracy focuses on what AI allows politicians to do: spread misinformation, manipulate citizens or conduct surveillance. Democracies have never relied only on laws and elections to check abuses of power.

Why it matters

A government doesn't need to remove humans altogether to defeat this safeguard. This is what makes the current generation of frontier AI especially concerning. The danger is compounded by what these technologies can do. In this regard, Anthropic has both a duty to warn democratic society about those risks and a duty to resist an autocratic demand that would contribute to them.

Not a licence to govern None of this means Anthropic should decide how AI gets governed.

What to watch

They've relied on people -- a great many of them, spread across a great many independent roles: the officer who can refuse an unlawful order, the engineer who can decline to build something, the analyst who notices an anomaly and says so, the journalist a whistleblower can call. People provide a safeguard. Democracies are resilient because responsibility is spread across many independently situated people.

The Cambridge Analytica scandal is a case in point. Facebook sat on evidence that user data had been improperly obtained and used by the political consultancy Cambridge Analytica. Such moments depend on there being enough independent people, in enough independent positions, for at least one of them to decide to speak. This is precisely what the autocratic use of AI threatens to dismantle. However, that doesn't fully address the problem.