Prompt Injection Benchmarks: What AgentDojo, BIPIA, and CyberSecEval Actually Measure

Meta's CyberSecEval 2 paper states the result in its abstract: "conditioning away risk of attack remains an unsolved problem; for example, all tested models showed between 26% and 41% successful prompt injection tests." The models tested were GPT-4, Mistral, Llama 3 70B-Instruct, and Code Llama. That range is the honest baseline for any deployment that treats the model itself as the defense against prompt injection.

I want to walk through the benchmarks that actually exist, what each one measures, where the numbers cluster, and what the adaptive-attack literature published since then does to every static score in the field.

The benchmarks that measure injection, and what each one holds

Prompt injection and jailbreak are different problems, and half the confusion in this space comes from conflating them. A jailbreak is the user attacking the model's safety policy. A prompt injection is third-party content hijacking a model that is acting on the user's behalf. JailbreakBench and HarmBench measure the former. The list below measures the latter.

| Benchmark | Publisher | Structure | Headline finding | |---|---|---|---| | AgentDojo | ETH Zurich, NeurIPS 2024 | 97 realistic tasks, 629 security test cases across email, e-banking, travel, workspace | Attacks break some security properties of tool-calling agents, and agents fail many tasks even with no attacker present | | BIPIA | Microsoft, KDD 2025 | Five task domains: email, web QA, table QA, summarization, code QA | LLMs are "universally vulnerable" to indirect injection | | InjecAgent | ACL Findings 2024 | 1,054 test cases, 17 user tools, 62 attacker tools, 30 agents evaluated | ReAct-prompted GPT-4 successfully attacked 24% of the time | | Open-Prompt-Injection | Liu et al., USENIX Security '24 | 5 attacks x 10 defenses x 10 LLMs x 7 tasks | The first formal framework; systematizes attack and defense definitions | | CyberSecEval 2 | Meta, Purple Llama | Injection plus code-interpreter abuse, with a false-refusal-rate counterweight | 26% to 41% success on every model tested | | Agent Security Bench | ICLR 2025 | 10 scenarios, 13 LLM backbones, 400+ tools, 27 attack and defense methods | Highest average attack success rate 84.30%; defenses show "limited effectiveness" | | MCPTox | 2025 | 45 real MCP servers, 353 real tools, 1,312 malicious cases, 20 agents | Attack success up to 72%; safety alignment gives minimal pre-execution protection |

Two crowdsourced datasets sit alongside these. Tensor Trust from UC Berkeley collected over 126,000 human-written prompt injection attacks and 46,000 defenses from an online attack-and-defend game, and the attacks generalized to deployed applications outside the game. Lakera's Gandalf the Red released 279,000 attacks from its gamified red-teaming platform and introduced D-SEC, a threat model that separates attackers from legitimate users across multi-step adaptive interaction.

One clarification worth making because it circulates as fact: NIST publishes no prompt injection benchmark. NIST's contributions are taxonomy and controls, principally AI 100-2e2025 on adversarial machine learning and the COSAiS control overlays. Anyone citing a "NIST prompt injection benchmark" is citing something that has never been published.

What the adaptive-attack literature did to every static score

A static benchmark measures a defense against a fixed attack set. An adaptive attacker rewrites the attack against the defense. Three papers, published across 2025, took published defenses and attacked them adaptively.

Zhan et al. took eight published indirect-prompt-injection defenses and broke all eight, with attack success rates consistently above 50%. NAACL Findings 2025.

A joint team from OpenAI, Anthropic, and Google DeepMind took twelve recent defenses and broke all twelve, most above 90% attack success, using gradient descent, reinforcement learning, random search, and human-guided search. Accepted to USENIX Security '26.

Google DeepMind, attacking its own production Gemini defenses, reported that the TAP adaptive attack succeeded in 99.8% of cases with near-zero customization and a few dollars of attacker compute. Baseline defenses such as Spotlighting and Self-reflection, which had looked strong against static attacks, "became much less effective." The paper's own conclusion is the sentence to keep: relying on defenses tested only against static attacks "offers a false sense of security."

Lakera's Gandalf work adds the cost side of the ledger. In-LLM defenses such as hardened system prompts degrade usability for legitimate users even in cases where they block nothing. The security and utility tradeoff is measurable rather than theoretical.

The architectural root cause

OWASP stated the mechanism plainly in its State of Agentic AI Security and Governance release in June 2026: large language models treat the system prompt, the user's request, and any text retrieved from external sources as a single stream of tokens, and there is no reliable way to mark some of those tokens as commands and others as data. OWASP's own mapping in the same document puts prompt injection at the root of 6 of the 10 categories in the Top 10 for Agentic Applications.

That is why every in-band defense has a ceiling. A filter that reads the prompt is reading the same undifferentiated token stream the model reads, and an attacker who can rewrite the input can rewrite the input to the filter too.

How to read a benchmark number before you deploy against it

A benchmark score is useful and it is not a control. Four questions turn a published number into an operational one:

1. Was the attack set static or adaptive? A static ASR of 3% and an adaptive ASR of 90% are the same defense.
2. Does the benchmark measure utility alongside security? AgentDojo does. A defense that blocks every attack by breaking the agent scores perfectly on a security-only benchmark.
3. Does the defense require model weights? BIPIA's near-zero result came from white-box adversarial training. That result cannot transfer to a deployment calling a closed model over an API.
4. What happens on the residual percentage? If a benchmark reports 4% attack success and the workload runs 250,000 model calls a month, the defense is letting through 10,000 successful injections a month. Design for what happens to those.

Question four is the one that determines architecture. I made the same argument from a different direction in model guardrails are not a security control.

Where the field went instead

The research response to the adaptive-attack results has been to move enforcement out of the model. Google DeepMind and ETH published CaMeL, which segregates control flow from data flow so untrusted content can never influence program flow, and enforces a capability system at tool-call time. It solves 67% of AgentDojo tasks with provable security properties rather than probabilistic ones.

The honest caveat, and the reason I will not oversell this line: a June 2026 paper, Adaptive Evaluation of Out-of-Band Defenses, points out that CaMeL, FIDES, Progent, RTBAS, and FORGE all report near-elimination of AgentDojo attacks while being validated only on static benchmarks, which is the exact methodology that made in-band defenses look strong before adaptive attacks broke twelve of them. Their own probe found Progent cut mean attack success roughly sixfold, from 25.8% to 4.2%, and a hand-crafted adaptive attack failed to raise it. The authors are careful to say this is consistent with, and does not establish, the hypothesis that out-of-band enforcement resists adaptive attackers.

The design principle survives the caveat. A control whose decision does not depend on the model correctly interpreting adversarial text has a different failure mode than one that does.

DeepInspect

This is the gap DeepInspect closes. DeepInspect sits at the AI request boundary as an external enforcement layer whose decisions are deterministic and identity-bound. It evaluates who is making the request, what role they hold, which route the traffic is on, and what data classification the payload carries, then passes, blocks, or modifies before the request reaches the model. The policy engine reads none of the adversarial text as instructions, because it is a policy engine rather than a language model.

The second half is the part benchmarks never measure. Every decision produces a per-decision audit record, signed, bound to identity, and committed before the response returns to the application. When the residual percentage fires and an injection does succeed, the record shows which principal's session it happened in, what the model was asked to do, which tools it reached, and what policy was active at that moment. Detection benchmarks tell you the rate. The audit record tells you the blast radius of the specific event, which is what an incident review and an EU AI Act Article 12 reviewer both ask for.

If you are sizing a prompt injection defense against a real traffic volume, let's talk today.