Is open source the answer to rogue AI agents? Nvidia's

What happened

AI agents are behind a steady string of security incidents this year. Nvidia thinks a new partnership built on open-source software is the answer. " The announcement is something of a response to Project Glasswing, the security alliance Anthropic spearheaded around its highly capable Mythos 5 model.

A model is truly open-source, however, when its code and training dataset are publicly accessible, meaning anyone could access its building blocks and understand more thoroughly how it works. Given how lucrative powerful proprietary models can be for companies like Anthropic and OpenAI, there are few incentives for fully open-sourcing a model.

Nvidia also pointed to the overall characterization of open-source AI as a possible hindrance for cybersecurity efforts going forward. "It will be crucial to recognize open models, harnesses, and security tooling as defensive assets, not liabilities, in AI and cybersecurity policy," the company said in the announcement.

Why it matters

Nvidia's initiative aims to further democratize AI security tools by focusing on open-source software rather than reserving access to a proprietary model for a few major companies. Nvidia said the Alliance was spurred by last week's Hugging Face incident, in which an OpenAI agent escaped a testing environment and infiltrated Hugging Face, stealing credentials and demonstrating what OpenAI said was an "unprecedented" outcome.

Nvidia argued in its announcement that because AI tools themselves have become an effective way to remedy AI attacks, open-source security tools must be a reliable public good. 2 model on its own infrastructure to analyze more than 17,000 actions and contain the intrusion," Nvidia noted.

Cloudflare, CrowdStrike, Adobe, IBM, the Linux Foundation, Microsoft, and ex-OpenAI executive Mira Murati's startup Thinking Machines Lab are among the first participants in the Alliance. Nvidia said its contribution to the effort will include new research on agent harnesses -- the infrastructure that turns an LLM into an agent by helping it act autonomously -- as well as open models, weights, and data.

In the announcement, Nvidia specifically argued for open models (alongside closed models) as a solution to security incidents, countering the dominant narrative that open-source AI can be more easily hijacked. "Some argue that open models are inherently less safe because they can be misused for cyberattacks or modified to remove guardrails," the company said.

" When it comes to AI, open-source means open-weight; that is, the model's final parameters and biases are public (rather than closed, as with Anthropic's or OpenAI's). Having that information, which shapes a model's outputs, lets developers fine-tune models for their own needs.

What to watch

" The Trump administration has been especially critical of open models from Chinese startups like Moonshoot and DeepSeek, which are often competitive with those from US labs, citing security risks. While those concerns are valid, they are also colored by worries that China will outpace the US in building the most sophisticated AI systems.

Nvidia's call to regulators also touches on the administration's increasing involvement in Anthropic and OpenAI's model release timelines reagrding security issues. 6.