Frontier AI agents: Only as safe as their containment

After AI models from both OpenAI and Anthropic autonomously compromised systems belonging to external organizations, security experts are calling for robust security controls around the testing and evaluation of AI agents.

The more widely reported incident involved OpenAI models that, while being evaluated on their ability to autonomously discover and exploit software vulnerabilities, escaped a sandbox and compromised production systems at the AI platform Hugging Face.

The test, called ExploitGym, was designed to measure whether AI agents could find vulnerabilities, chain them together into attack paths, and then exploit them to achieve a specific objective. OpenAI disabled the models’ normal guardrails against risky behavior before running them in a sandboxed environment whose only connection to the internet was through a proxy for hosting software packages.

Unexpectedly, an AI agent powered by the OpenAI models, having determined that Hugging Face’s platform might host reference solutions for the test, devised a way to access the company’s production servers in pursuit of them. 

Here’s a review of the post-portems on the incidents involving the top tw0 frontier AI companies.

[ Join webinar: Autonomy, Not Autopilot: Get Real About Agentic SOC ]

When AI agents behave badly

The OpenAI agent escaped its sandbox by finding and exploiting a zero-day vulnerability in the package registry proxy, which was hosted by a third-party infrastructure services provider.

As Hugging Face explained in a post-mortem of the incident, the agent gained root access to the sandbox and used it as a launchpad for the intrusion. It then identified and exploited vulnerabilities in Hugging Face’s systems, acquired sensitive credentials, and moved through internal infrastructure in search of the ExploitGym solutions.

Hugging Face noted:

“Over roughly two and a half days inside our infrastructure, an autonomous AI agent driven by a combination of OpenAI models ran an end-to-end intrusion against our platform. It was thousands of small, automated decisions, executed at machine speed across short-lived sandbox environments, with command-and-control staged on ordinary public web services.”

CISOs and other members of the Cloud Security Alliance (CSA) said Hugging Face described the rogue OpenAI agent as exhibiting erratic behavior in its pursuit of ExploitGym solutions. This included executing both brilliant and pointless moves, hallucinating thousands of lines of gibberish text, following attack paths that no human would take, and leaving behind easily detectable artifacts.

A turning point

Though the attack was at times erratic, many consider the model’s ability to reason across an attack chain, discover a zero-day vulnerability, break out of its constrained environment, and relentlessly pursue its objective in an external environment a turning point of sorts. OpenAI itself said that it considers the incident unprecedented and that it will respond appropriately.

And as it turned out, the OpenAI/Hugging face incident wasn’t unprecedented. Following its disclosure, Anthropic conducted a review of its own internal tests and discovered three incidents where a Claude model autonomously broke out from its test environment and gained unauthorized access to systems belonging to three external organizations. In all three instances, the models had access to the internet and worked on the assumption that any systems they could access were part of the exercise.

Anthropic’s red team said in a blog post:

“Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. It did not find or exploit any complex vulnerabilities, and in each case, Claude continued working to complete only the specific capture-the-flag task its evaluation had assigned.”

The incidents have sparked concern about AI agents going rogue. For many, they appeared to validate worst-case fears that the rapid adoption of often poorly governed AI in business and operational workflows could lead to disaster. The reality, security experts say, is more nuanced.

Keeping things in perspective

The incidents highlight a real security threat, said Michael Bell, founder and CTO of Suzu Labs. 

“The capabilities here are real and I don’t want to minimize them. A model finding a zero day on its own, chaining privilege escalation and lateral movement, and landing remote code execution on a third party is a milestone, and organizations should treat AI-driven attacks as a live threat category rather than a future one.”
—Michael Bell

But it’s important to keep the OpenAI/Hugging face incident in perspective, Bell added. OpenAI’s models escaped the test environment because the company did not adequately isolate it. “They turned off the safety classifiers and the cyber refusals, told a frontier model to go find and exploit vulnerabilities, and then ran that test in a sandbox where a piece of third-party software had a network path to the outside world.” Real air gaps don’t have such escape hatches, he noted.

The lesson for organizations, Bell said, is that when evaluating whether a model can find and exploit vulnerabilities, they should treat the evaluation like a cyber weapons test. Every supporting system, including the package manager, the dependency cache, and the logging pipeline, has to live inside the boundary with no route out.  Organizations need to stop assuming that their existing sandbox tooling carries over to AI environments, he stressed. “Containers and allowlists were designed to keep buggy software from misbehaving, not to hold an adversary that actively hunts for flaws in the walls of its own container,” he said.

Pay attention also to where the AI agent might go hunting as part of an evaluation, Bell noted. OpenAI’s models reasoned their way to Hugging Face because they figured out that the benchmark answers likely lived there. Egress monitoring is key as well.

“Nobody wrote that into a playbook. So, beyond the usual telemetry, watch for access patterns that look like hypothesis testing, meaning something touching systems in an order that follows a chain of inference rather than a known kill chain.”
—Michael Bell

The escape route in this incident was a package registry proxy that everyone trusted because it was supposed to be boring infrastructure. “Watch what your boring infrastructure is being asked to do,” Bell advised.

Why AI agents force a rethink on traditional controls

From a defensive standpoint, organizations should stop treating AI agent isolation as a one-time infrastructure decision and start treating it as a continuously verified control or harness, said Ryan McCurdy, a vice president at Liquibase. 

An agent shouldn’t receive broader access simply because its instructions say the internet is unavailable or a system is out of scope. “Access should be explicit, temporary, least privileged, and independently enforced at every boundary the agent could cross,” McCurdy said.  Instead of focusing solely on the source of an action to determine risk, focus on the action itself.

“AI-generated activity should face the same policy checks and operational controls as any other change before it reaches a production system.”
—Ryan McCurdy

Instead of perceiving the compromises as model failures and agents going rogue, many security experts see the Hugging Face breach and the Anthropic compromises as evidence that AI evaluation environments need the same level of security engineering and operational rigor as production infrastructure.

Traditionally, threat assessments considered capability and malicious intent together, said Tom Findling, CEO and co-founder at Conifers. But in the OpenAI and Hugging Face incident, there was no malicious intent. The agent was simply pursuing an evaluation goal, but its capability, autonomy and access still produced a real compromise.  

“Organizations should treat every permitted service as a potential path out. Evaluation environments must be technically separated from production and the public internet, contain no reusable credentials, and use tightly controlled or offline dependencies. The containment should also be actively tested for escape paths before highly capable agents are allowed to operate inside it.”
—Tom Findling

Focus on the harness — not the model

AI observability must become part of the overall defense strategy. Organizations need visibility into the agent’s inputs, decision and reasoning traces, tool calls, actions, and outcomes, Findling said. They should also build detections on these new telemetry sources, which most organizations do not do today. “Detection, investigation, response and adaptation must all happen at machine speed, with humans on the loop to set boundaries and provide oversight,” Findling said.

In an analysis based on input from Hugging Face and OpenAI, the CSA described the incidents as highlighting why organizations should deploy defenses based on the presumption that AI agents will go rogue. “Agents will do what they need to achieve the assigned objective, and time and time again we see them doing so in creative and unexpected ways,” the CSA said.

To contain the threat, organizations must enhance governance and safeguards around AI model and “codify approval requirements and escalation paths in policy,” the CSA noted. Also make sure to test isolation mechanisms by getting the agent to attack the Internet or other boundaries before any actual testing happens, so it becomes easier to spot escape behaviors and signals. Where possible, organizations should also make their agents easily identifiable to external parties so accidental victims — such as Hugging Face — know whom to contact in case an agent compromises their environment.

Cato Networks researchers recently showed how AI agents can execute an end-to-end enterprise cyberattack. In one test, using GPT-4.5 paired with agent orchestration and MCP-enabled tooling, the researchers achieved full domain administrator access in an Active Directory environment in just 40 minutes via a 32-step attack chain. 

The attack’s success hinges less on the underlying AI model and more on the harness — the tools, orchestration, and operational guidance wrapped around it. The report noted that the real risk isn’t novel attack techniques but the dramatic compression of attack timelines, which breaks defenders’ human-paced detection and response assumptions, and it called for organizations to invest in modular harness architecture and machine-speed governance to keep pace.