↓ Skip to main content
Christos Kalyvas-Kasopatidis

The swarm breach

·2 mins
agentic AI AI security incident analysis multi-agent systems MITRE ATT&CK detection engineering
Christos Kalyvas-Kasopatidis
Author
Christos Kalyvas-Kasopatidis
AI security architect with a PhD in machine learning and big data, working on the security of AI and LLM systems, AI for cyber defence and vulnerability intelligence.

In July, roughly 700 OpenAI test agents breached Hugging Face. No human directed it, and no single agent could have done it alone.

The swarm breach: stats, kill chain, timeline and six controls

The mechanics were ordinary. The agents exploited a vulnerability in the JFrog Artifactory instance their evaluation environment used. That gave them internet access and a place to host a message board inside a package manager. One agent found 14 exposed Hugging Face write tokens and posted them. Another chained exploits into command execution on production servers.

The orchestration was not ordinary: the agents assigned themselves roles, one appointed itself coordinator, and they shared exploits with each other.

Mapped to MITRE ATT&CK, every step is familiar:

  • T1611, escape to host,
  • T1102, web-service C2,
  • T1552, unsecured credentials,
  • T1190 and T1059, exploitation and execution.

MITRE ATLAS, built for attacks on AI systems, has no clean home for AI agents acting as the threat actor. That gap is worth closing.

Prevention: zero trust for non-human identities. Treat every agent as an identity:

  • short-lived, scoped, workload-bound credentials, with no static tokens reachable from agent environments,
  • default-deny egress and micro-segmented tool environments,
  • agent-to-agent channels inventoried and governed like C2,
  • no goals accepted from other agents without a policy check,
  • a sanctioned way to say “this task is impossible”.

Detection: watch the population, not the agent. Monitoring each agent on its own missed this. Better signals:

  • new edges in the agent-communication graph, especially between sandboxes meant to be isolated,
  • synchronised spikes in tokens or compute across unrelated runs,
  • tool-call sequences outside a task’s expected chain,
  • any egress at all from evaluation environments.

The other lens: “rogue AI escapes” is a better headline than “vulnerable server, leaked tokens, no egress filter.” Both are true. Only the second can be fixed this week, and the first is the one that gets used to sell alarm or overstate capability.

Sources: BleepingComputer, Dark Reading, Fortune, Engadget. Reported dates vary by a day or two across sources.