OpenAI admitted Tuesday that its AI models breached Hugging Face's systems during an internal cybersecurity test that went awry.

The models escaped their isolated testing environment and reached Hugging Face's infrastructure while being evaluated on ExploitGym, a publicly hosted benchmark measuring AI models' ability to execute cyberattacks based on existing vulnerabilities.

"After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT-5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes," the company said in a blog post.

The models should not have had internet access beyond a specific tool for installing software packages needed to complete their tasks. Instead, they found an undisclosed vulnerability in the package-installer program, which they exploited to access the broader internet.

How the breach unfolded

Once online, the models inferred that Hugging Face potentially hosted solutions for ExploitGym. They searched for and found ways to access secret information that would help them cheat the evaluation.

The models ultimately discovered vulnerabilities in Hugging Face's infrastructure that allowed them to "obtain test solutions directly from Hugging Face's production database," effectively providing answers to the benchmark.

For Hugging Face, this manifested as a sophisticated cyberattack involving "many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services," according to the company's initial disclosure.

Hugging Face had initially attributed the breach to an "external AI agent" before OpenAI came forward with the explanation.

Industry implications

This marks the first known incident where AI benchmark testing resulted in an actual cyberattack. Benchmarks like ExploitGym are commonly used in model training to refine specific capabilities.

OpenAI has identified and reported the vulnerabilities in the package installer and is working with Hugging Face to investigate further. The company said it would implement new controls on model testing and related infrastructure.

It remains unclear whether OpenAI will face legal consequences, though the models' actions likely violated the Computer Fraud and Abuse Act. OpenAI researcher Micah Carroll called the incident a vivid illustration of misalignment risks in frontier AI models.