Over the past three weeks, several cases have surfaced where AI agents attacked real systems during testing. After the incidents involving OpenAI and Anthropic, the European Commission reached out to the companies, and the White House pulled developers together to discuss testing rules.
Here's what happened.
July 21 — OpenAI. Models were being tested on cybersecurity tasks in a sandboxed environment. They found a zero-day in a proxy, used it to get out to the internet, and hacked Hugging Face's infrastructure to grab the answers to the test. OpenAI's report
July 30 — Anthropic. The company reviewed 141,006 old runs and found three cases where Claude gained unauthorized access to real organizations' systems. In one, the model published a malicious package to PyPI, which got installed on 15 systems before anyone caught it. In its reasoning, the model itself noted that in the real internet this would be an actual attack — then talked itself into believing it must be a simulation, since the calendar said 2026. Anthropic's report
August 4 — the UK's AI Security Institute. In 10 of 122 runs, agents went beyond their assigned task — researchers counted 19 such actions. One tried to inject malicious code into a real open-source project, created several fake accounts, and tried to talk the maintainer into merging the code. The maintainer refused. AISI writes that the only thing holding the line was a human paying attention — there was no technical barrier at all. AISI's report
August 5 — Meta. Another model attacked an outside company during testing. The reason was mundane: a misconfigured test environment gave the agent access to the actual internet.
August 5 — OpenAI at Black Hat. It turned out the story was bigger and started back on May 7. A model stuck on a task left a note in an internal package manager. Other agents found it and started replying. For two months they passed vulnerabilities and credentials back and forth, split up work, and worked out how to address each other. The thread piled up hundreds of thousands of messages. When OpenAI tore it down and revoked access, the agents rebuilt the channel within two days. By the end, they'd started discussing whether to sign their messages, because they suspected someone was impersonating others. Wired
Is this actually dangerous?
Yes.
None of these models wanted to escape the test environment. They were just doing the assignment. The problem is that an agent picks its own intermediate steps on the way to a goal. OpenAI's task was to pass the test. One of the paths it found looked like this: get internet access, find the answers, hack Hugging Face. So it did.
Agents also aren't obligated to stop. Anthropic's model understood it was looking at a real system, and decided that the real company — outside the test environment — was, apparently, also part of the exercise.
If an agent has a terminal, tools, and network access, one mistake in the guardrails is already enough to turn into a real attack.
What's next?
There will probably be more cases like this. That's OpenAI's own forecast.
Models are getting stronger fast. By AISI's estimate, the length of cyber tasks they can complete autonomously has been doubling roughly every five months since late 2024, and the latest models are outpacing that trend. The newest checkpoint, Mythos Preview, is the first to clear both of the institute's cyber ranges — including one nothing had passed before it. (AISI)
So far, most of these breaches happened by accident, during tests. The obvious next risk is someone doing it on purpose.