Open AI says its AI model “went rogue”: What do we know? 

OpenAI has revealed that one of its artificial intelligence models independently stole login credentials and hacked into another technology company’s system, in what is widely seen as one of the first known incidents of AI systems acting autonomously.

“We had a significant security incident during evaluation of our models,” CEO Sam Altman posted on X on Tuesday.

The incident comes as calls mount from technology rights advocates for stricter guardrails on rapidly evolving AI systems.

They have grown so powerful in a short span of time that alarming phenomena such as deepfakes and sophisticated cyberscams are becoming the norm.

Earlier this year, a number of software engineers quit their jobs at top companies such as Anthropic and AI in protest against how the technologies are being built.

“AI is accelerating the discovery and exploitation of vulnerabilities,” OpenAI said in a lengthy statement on Tuesday that detailed the latest incident.

“The primary lesson from this incident is that model security and safety must keep pace with rapidly advancing capabilities.”

Here’s what we know about the breach:

What has happened?

OpenAI said two of its models found their way out of an isolated, no-internet access environment – or a sandbox – and hacked into the systems of tech company Hugging Face on their own.

The models involved are the latest GPT-5.6 Sol model and an unreleased model the company said is “even more capable,” than its latest version.

Hugging Face hosts openly sourced AI models and resources. The two OpenAI agents discovered vulnerabilities in Hugging Face’s servers and proceeded to steal login details and then hack into the company’s systems.

The incident occurred during an OpenAI internal testing session designed to assess the models’ cybersecurity capabilities. OpenAI had removed standard safety measures for the test.

Both sought to cheat their way through a problem during the test, OpenAI said. They went to “extreme lengths to achieve a rather narrow testing goal” and “found ways to gain access to secret information that it could use to cheat the evaluation”.

OpenAI’s security team detected the unusual activity internally, but details of the breach came to light following a joint investigation by both companies.

What has Hugging Face said?

Hugging Face disclosed last Thursday that its servers were hacked by an unknown but sophisticated agent acting on its own. The company discovered the breach through its own AI-assisted detection.

“This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system,” the company said.

Following OpenAI’s disclosure that its models were involved in the breach, both sides conducted an ongoing joint investigation this week.

“We suspected last week’s cyberattack might have come from a frontier lab, given the sophistication of the agent. Turns out it did!” CEO Clement Delangue posted on X on Tuesday.

Hugging Face’s staff “strongly believe there was no malicious intent on their part,” Delangue added, referring to OpenAI.

Why does this matter?

Cybersecurity experts have previously sounded the alarm over the potential, extreme capabilities of AI systems and the dangers they pose.

But until now, there have been few real-life cases proving those concerns like this one.

Many warn that incidents like these could become commonplace and that AI systems pose a threat to financial, security and other sensitive data systems.

OpenAI revealed in a separate incident earlier this week that the unreleased, more powerful model had escaped an isolated environment during another test.

Anthropic, OpenAI’s rival, had similar issues with its most powerful agent to date, the Claude Mythos Preview model.

During a stress test of an early version, the model found its way out of a sandbox, gained internet access and emailed the supervising researcher that it had escaped and then wiped evidence of its activity. Anthropic halted a planned public release of the model afterwards.

In April, the US Federal Reserve and the Treasury Department convened a meeting with bank CEOs where officials warned about the cybersecurity risks posed by Mythos. Canada’s federal banking regulator has also warned financial institutions about the model’s capabilities.

The OpenAI breach also appears to make the case for companies like Hugging Face, which rely on open source systems, as opposed to more secretive AI development platforms like OpenAI.

“This incident, possibly the first of its kind, proves a point we’ve long believed: AI safety won’t be solved by any single company working in secret,” Hugging Face’s Delangue was quoted as saying in OpenAI’s statement.

“It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere,” he added.