OpenAI claims responsibility for the Hugging Face hack

What happened

During an internal security evaluation, OpenAI models escaped their sandbox, independently discovered a zero-day vulnerability, and breached Hugging Face's production infrastructure. " Both ran with reduced security filters to test their maximum cyber capabilities. Hugging Face had previously made the incident public after detecting and containing the activity on its infrastructure. The incident happened during an internal evaluation using the ExploitGym benchmark, which challenges models to follow complex attack paths.

Hugging Face had already started a forensic reconstruction using its own open-source models when the two companies joined forces. The company said it had to rely on open models because proprietary models refused cyber-related prompts due to safety guardrails. OpenAI says the incident shows that theoretical capabilities to autonomously carry out full cyberattacks hold up in real-world environments.

It reported the zero-day to the affected third-party provider, and a patch is in development. Hugging Face is now part of OpenAI's Trusted Access Program. The question is how much PR spin is baked into this. There are a few reasons to take it seriously. Hugging Face confirmed the incident independently. The company has no reason to prop up OpenAI's narrative.

It has its own open-source agenda and would gain nothing from playing along with a fabricated story. Hugging Face co-founder Thomas Wolf says, "This incident also reinforced my belief in the importance of access to capable open-weight models for cyber defense. " There's als

Why it matters

The production classifiers that normally block high-risk cyber activity were intentionally turned off for the test. The models ran in an isolated environment where network access was limited to installing packages through an internally hosted proxy. OpenAI says the models burned significant inference compute to find a path to the open internet. They discovered and exploited a zero-day vulnerability in the package registry cache proxy.

OpenAI has since reported the flaw to the affected provider. From there, the models carried out a series of privilege escalations and lateral movements within OpenAI's research environment until they reached a node with internet access. The reason is almost absurd. The models independently concluded that Hugging Face might host models, datasets, and solutions for ExploitGym.

In one instance, a model chained together multiple attack vectors, including stolen credentials and zero-day vulnerabilities, to find a remote code execution path on Hugging Face servers. The models tried to pull test solutions directly from Hugging Face's production database so they could cheat during the evaluation.

"All evidence suggests that the models were hyper-focused on finding a solution for ExploitGym and employed extreme measures to achieve a fairly narrow testing goal," OpenAI writes. OpenAI's security team spotted the anomalous activity internally. Hugging Face security personnel and their own AI agents detected and shut down the activity on their infrastructure simultaneously.

What to watch

The UK AI Safety Institute and others previously measured these capabilities in benchmarks. Advanced models can discover and exploit novel attack vectors in production systems without access to the source code. OpenAI acknowledges that intentionally disabling security filters during evaluation was an inadequate practice. The company will tighten security measures for future training and evaluations and has implemented stricter controls on infrastructure configuration until the vulnerabilities are patched.