OpenAI's Hugging Face breach exposes AI's next safety challenge

Frontier AI models are getting scary good at breaking rules in ways their creators didn't anticipate.

Why it matters: Forget AGI and superintelligence timelines. Today's models are already slipping past guardrails, carrying out sophisticated, multistep cyberattacks and — in at least one case — compromising real-world infrastructure, sometimes before their creators know what happened.

Case in point: OpenAI said Tuesday that GPT-5.6 Sol and "an even more capable pre-release model" carried out last week's AI-led cyberattack on Hugging Face.

• OpenAI says its models were asked to solve a hacking challenge during pre-deployment testing and went to extreme lengths to win.
• The models decided on their own to break out of their walled testing environment, inferring that Hugging Face — a popular platform for hosting AI models and datasets — might hold the test's answers.
• The models used stolen credentials and additional vulnerabilities to gain access to part of Hugging Face's production infrastructure.

What they're saying: Clément Delangue, co-founder and CEO of Hugging Face, called the incident an "attack unlike anything we've seen before" and praised OpenAI for its partnership as the companies investigate what happened.

• "It's quite mind-blowing that all of this happened autonomously," he added.
• Logan Graham, head of Anthropic's frontier red team, said he told his team to "remember this moment as the first true AI safety incident."

The intrigue: Hugging Face used GLM 5.2, an open-weight model from Chinese AI company Z.ai, to analyze the attack after running into guardrails when using U.S. frontier models.

Between the lines: OpenAI's latest models aren't the only ones finding ways to cheat evaluations.

• The U.K.'s AI Security Institute said Tuesday that every model it tested attempted to cheat at least some of the time on its cybersecurity evaluations.
• AISI defines cheating as taking an out-of-scope or explicitly prohibited action to achieve the task's goal.
• GPT-5.6 Sol attempted to cheat in 12.6% of test runs, while Anthropic's Claude Mythos Preview did so in 7.8%.
• Models often failed to admit they had cheated when questioned afterward and described their cheating as wrong only less than half the time.

Zoom in: Xbow — whose autonomous AI agents probe clients' systems for security holes, with permission — said Wednesday that it has seen its own agents do similar things in internal testing.

• Seven months ago, the company forgot to switch on its safety guardrails during a lab test. Its agent then broke into a system, stole credentials and used them to map the target's Slack workspace and probe its AWS accounts.

Threat level: It isn't new for models to game their safety evaluations. But as models grow more powerful, the fallout from these shortcuts is getting more severe, Chris Canal, CEO and co-founder of third-party evaluation company EquiStamp, told Axios.

• "Letting your model loose on the internet has a blast radius," Canal said. "If anything goes wrong, it could be hugely impactful, maybe to people's lives."
• Canal was speaking generally about internet-connected AI evaluations, not OpenAI's specific incident.

The big picture: The most capable OpenAI model behind the Hugging Face breach isn't even public yet, raising the question of how safety testing needs to adapt to keep pace.

• Canal said independent evaluators previously had about five weeks to test a pre-release model before launch. That window has shrunk to as little as five days as companies race to ship.

Reality check: The versions of these models the public can use carry stronger safeguards designed to block Hugging Face-style attacks.

• OpenAI, like other companies, intentionally dialed back those cyber safeguards for GPT-5.6 Sol and its unreleased model inside the testing environment — making them far more capable hackers.