How is a testing program different from a red team engagement?

A red team engagement is a point-in-time assessment by skilled humans who probe creatively and report what they found. A testing program is an automated, repeatable suite that runs on every change and reports a tracked metric. They are complementary. The red team finds novel attacks and application-specific weaknesses a corpus would miss; the program encodes those findings as regression cases so the same bug cannot silently return on the next deploy. Run the red team periodically and the program continuously.

What attack success rate threshold should fail the build?

There is no universal number, because attack success rate is relative to your corpus and your objectives. A practical approach sets the gate low, in the low single digits of percent, and treats any increase over the previous build as a regression to investigate. The absolute value matters less than the trend: a suite that passed at 1% last week and reports 8% this week is telling you a recent change opened a path, regardless of where you set the line. Tighten the gate as the corpus matures.

Should I test the model or the whole application?

The whole application path, including retrieval, tool wiring, and system prompt assembly. Most real prompt injection exposure lives in orchestration rather than the bare model: indirect injection arrives through retrieved documents and tool outputs, and unauthorized tool use depends on how your app wires tools to the model. Testing the raw model misses exactly the surface attackers use. Send each case through the same code path a real request travels, so the success check reflects what the application would actually do.

Does passing the suite mean we are safe from prompt injection?

No, and a program that claims otherwise is misreading its own metric. Passing means the application resisted the attacks in your corpus on this build. Adaptive attackers tune to your specific defenses, and published research shows most defenses fall under that pressure, several at well over 90% success. Testing measures exposure to enumerated attacks and catches regressions; it cannot certify resistance to an attacker who has not been enumerated. Treat a passing suite as necessary hygiene and pair it with a runtime enforcement layer that does not depend on the model refusing.

Prompt Injection Testing: Building a Repeatable Program Instead of a One-Off Red Team

A prompt injection test asks one question: can crafted input override the instructions the application gave the model. Answering it once, by hand, tells you the state of one build on one afternoon. The application ships again next week with a new system prompt, a new tool, or a new retrieval source, and the answer can flip. Meta's CyberSecEval 2 benchmark (arXiv:2404.13161) found successful prompt injection across every model it tested at rates between 26% and 41%, which means the baseline exposure is high enough that spot-checks are not a control. I want to walk how to build prompt injection testing as a repeatable program, wired into the pipeline, measuring a defined objective, and honest about the ceiling that a fixed corpus cannot see past. It builds on prompt injection test cases and prompt injection defense in depth.

Define the objective before you write a single test

An attack succeeds or fails only against a stated goal. Testing without a defined objective produces a pile of model outputs nobody can grade. Pick concrete, checkable objectives for your application:

Exfiltration: the model reveals the system prompt or a value it was told to keep secret.
Unauthorized tool use: the model calls a tool, or supplies arguments, the caller was not permitted to trigger.
Output manipulation: the model emits attacker-chosen content into a downstream system that trusts it.
Policy bypass: the model produces content the application is required to refuse.

Each objective needs a programmatic success check. "The model said something bad" is not gradable; "the response contains the canary string CANARY-7F3A" is.

Assemble a corpus across both vectors

Prompt injection arrives through two paths, and a corpus that tests only one misses half the surface. Direct injection puts the attack in user input. Indirect injection plants it in content the model retrieves: a web page, a document, an email, an MCP tool description. The indirect vector is the one that matters most for agents, because the agent ingests untrusted content as a normal part of its job.

Structure the corpus as data, not code, so it grows without touching the runner:

Seed it from public sources like the OWASP LLM Top 10 entry on prompt injection and MITRE ATLAS technique records, then add cases specific to your tools, your retrieval sources, and your refusal requirements. The application-specific cases are the ones that find real bugs.

Wire the runner into CI

A corpus that runs when someone remembers is not a control. The runner loads the corpus, sends each case through the real application path, evaluates the success check, and fails the build when the attack success rate crosses a threshold:

Two design rules keep the signal honest. Exercise the full application path, including retrieval and tool wiring, because testing the bare model misses the injection surface that lives in your orchestration. And run it on every change to the system prompt, the tools, the retrieval sources, or the model version, because any of those can move the number.

Measure attack success rate, and read it correctly

Attack success rate, the fraction of corpus cases that hit their objective, is the metric. It is a relative measure against your corpus, so track it as a trend across builds and treat a rising ASR as a regression. The number a single run reports is a floor on your exposure and never a ceiling, because your corpus is finite and attackers are not.

The adaptive-attack ceiling

Here is the limit every testing program has to state plainly. A fixed corpus measures known attacks. Adaptive attackers tune to your specific defense, and the research on this is stark: a Google DeepMind study (arXiv:2505.14534) reported a tree-of-attacks method reaching 99.8% success against production prompt-injection defenses once the attack adapted to them, and Nasr et al. (arXiv:2510.09023) bypassed twelve published defenses, most at over 90%, under adaptive pressure. A passing test suite means your application resists the attacks in your corpus. It does not mean the application is safe against an attacker who studies it. Testing quantifies exposure to known patterns and validates that defenses did not regress. It cannot promise resistance to an adversary who has not been enumerated, which is why the deployment posture cannot rest on the model refusing.

DeepInspect

Testing tells you where prompt injection lands. Closing the exposure is a runtime problem, and that is where DeepInspect sits.

DeepInspect is a stateless proxy on the AI request and response path. It does not depend on the model refusing an injected instruction, which is the assumption the adaptive-attack research keeps breaking. It binds every request to the caller identity the application supplies, authorizes each model call and each tool invocation against role and content, and evaluates the response before it reaches a downstream system, applying a deterministic pass, block, or modify decision inline and fail-closed. When an injection convinces the model to attempt an unauthorized tool call or to emit data it was told to protect, the enforcement layer is the control that stops the action rather than the corpus that predicted it. Keep the testing program to measure exposure, and put an external enforcement layer on the path to hold it. Book a demo today.