← All lab updates
An attacker that learns: why a defence has to hold when the attack adapts
Many published defences against hidden commands look strong against a fixed list of attacks and collapse when the attacker adapts. We tested our correction, which lives in the model's weights, on the public agent benchmark and against a second model that rewrites its attack after every failure. The numbers, what still gets through, and why adaptive testing should be the standard.
## Why fixed tests flatter
A defence tested against a fixed list of attacks only proves it stops that list. A real attacker reads the refusal and tries again. A recent study showed one of the best-known published defences going from about **1 success in 100 to about 7 in 10** once an attacker was trained against it.
## The two tests
1. **The public agent benchmark** (AgentDojo): 97 everyday tasks in banking, messaging, travel and office work, and 949 versions with a command planted in the data the agent reads.
2. **An attacker that adapts.** A second model writes the hidden text, sees the reply, and rewrites it, up to ten times per case, keeping a library of what worked. It practised on 310 cases, then attacked 100 it had never seen.
The original and the corrected model ran on the same server; the only difference is the correction in the weights.
```chart
{"title": "Hidden commands that took over the agent", "unit": "%", "series": ["Before", "After the correction"], "rows": [["Public agent benchmark", 8.8, 0.4], ["Attacker that adapts, 10 tries", 68, 8]], "note": "Share of cases where the agent followed the hidden command. Lower is better.", "max": 70}
```
- **Benchmark:** about 9 in 100 before, about 4 in 1,000 after, roughly twenty times fewer.
- **Adaptive attacker:** about 7 in 10 before, under 1 in 10 after.
- **The agent still works:** it completed the user's tasks about as often as before.
## What still gets through
The few attacks that worked pretend that a broken system needs the dangerous action before the user's task can finish, or ride on a task where the user already asked for a payment. Naming them is the next round of the correction. Full write-up: [the correction holds against an adaptive attacker](/lab/correction-holds-against-adaptive-attacker).
## What to ask of any defence
1. **Show results against an attacker that adapts,** not only a fixed list.
2. **Show the cost:** does the protected system still do its job?
3. **Show what still gets through.** A defence with no known failures has not been tested hard enough.
This update is not available.