How is LLM security testing different from a general application pen test?

The failure classes are different. A general pen test looks for injection, broken authorization, and data exposure in code and infrastructure. LLM security testing measures model-specific behaviors: whether crafted input overrides instructions, whether the model can be jailbroken out of refusals, whether sensitive context leaks into output, and whether the application emits unsafe content on its own. The metrics are behavioral rates rather than binary vulnerability findings, and the test cases are prompts and contexts rather than payloads against endpoints. You want both, because an LLM application still has a conventional attack surface underneath its model-specific one.

What is the difference between injection and jailbreak testing?

Target. Injection testing measures whether crafted input can override the instructions your application gave the model, such as forcing an unauthorized tool call or revealing the system prompt. Jailbreak testing measures whether the model can be talked out of its own safety training through role-play, obfuscation, or multi-turn manipulation. They overlap in technique and differ in what they defeat: injection defeats your application's control over the model, and jailbreak defeats the model's refusal behavior. Test both against your application's specific requirements rather than generic ones, because your refusal and authorization rules are narrower than the provider's defaults.

How do canary values help test data exfiltration?

They turn a judgment call into a binary check. Place a marked secret, a canary, into the sensitive context the model can see, then measure whether it appears in the output. If the canary escapes, exfiltration happened, and no human grading is required to know it. This makes exfiltration exposure a trackable number across builds and a clean CI gate, where grading whether real content leaked would need a reviewer. Use distinct canaries per context source so a leak also tells you which source path exposed it.

Does passing a testing program make the application safe?

No, and reading it that way misuses the result. A passing program means the application resisted the attacks in your corpus and did not regress since the last build. Adaptive attackers tune to the specific deployment, and published research shows most defenses fall under that pressure, several above 90% success. Testing measures exposure to enumerated attacks and prevents regressions; it cannot promise resistance to an adversary who has not been enumerated. Treat a passing program as required hygiene, track the trend rather than the absolute pass, and pair it with a runtime enforcement layer that does not rely on the model resisting.

LLM Security Testing: The Four Categories and What Each One Measures

Security testing for a traditional service checks a known set of failure classes: injection, broken authorization, data exposure. An LLM application has its own set, and testing it means measuring four distinct things that most teams collapse into a single vague "we tested the model." The four are prompt injection resistance, jailbreak resistance, data exfiltration exposure, and output safety, and each answers a different question against a different metric. Meta's CyberSecEval 2 benchmark (arXiv:2404.13161) found successful prompt injection between 26% and 41% across every model it tested, and the Agent Security Bench study (arXiv:2410.02644) reported peak average attack success rates above 84% against agents, so the baseline exposure is high enough that testing has to be structured rather than impressionistic. I want to define each category, the metric it reports, and how they fit into a repeatable program. The prompt-injection category has its own build-out in prompt injection testing; this piece is the wider map.

Category one: prompt injection resistance

The question: can crafted input, direct or indirect, override the application's instructions to the model. The objective is a concrete override, such as revealing the system prompt or triggering an unauthorized tool call, and the metric is attack success rate against a corpus of injection cases. Indirect injection, planted in retrieved content rather than user input, is the category that matters most for agents and the one a naive test skips. Seed the corpus from the OWASP LLM Top 10 and MITRE ATLAS and extend it with cases specific to your tools and retrieval sources.

Category two: jailbreak resistance

The question: can the model be talked out of its safety behavior. Jailbreaking overlaps with injection and is distinct in target: injection overrides the application's instructions, and jailbreaking defeats the model's own refusal training through role-play, obfuscation, or multi-turn setups. The metric is again attack success rate, measured against your application's refusal requirements rather than the model's generic ones, because what your app must refuse is narrower and more specific than the provider's defaults. The distinction from injection is covered in prompt injection vs jailbreak.

Category three: data exfiltration exposure

The question: can the application be made to emit data it holds but should not reveal. This tests the path from sensitive context, retrieved documents, system prompt, or tool output, to the model's response. The objective is the model returning a value it was supposed to protect, and the metric is the rate at which planted canary values escape into output. A canary is a marked secret placed in context whose appearance in a response is an unambiguous, programmatically checkable failure:

Canary-based testing gives an exact number where content grading would need a human, which is what makes exfiltration exposure trackable across builds.

Category four: output safety

The question: does the application emit unsafe, non-compliant, or downstream-dangerous content even without an attacker. This covers hallucinated actions, unsafe code, disallowed content, and output that a trusting downstream system would execute. The metric is a violation rate against a set of prompts drawn from real usage plus adversarial edge cases. Output safety is where testing meets the LLM security checklist, because a failed output-safety case often maps to a missing control rather than a clever attack.

Wiring the four into one program

Each category is a suite of cases with a programmatic success check, and together they form a runner that executes on every meaningful change and gates the build on a combined threshold:

Run the program on any change to the system prompt, the tools, the retrieval sources, or the model version, and exercise the full application path rather than the bare model, because most exposure lives in the orchestration around the model.

The ceiling testing cannot cross

Every category shares one limit. A finite corpus measures enumerated attacks, and adaptive attackers tune to your specific defense. The research is direct: a Google DeepMind study (arXiv:2505.14534) reported an adaptive method reaching 99.8% success against production prompt-injection defenses, and Nasr et al. (arXiv:2510.09023) bypassed twelve published defenses under adaptive pressure, most at over 90%. A passing program means the application resisted the attacks you enumerated and did not regress. It cannot certify resistance to an attacker who studies the deployment. Testing quantifies exposure and catches regressions, and closing the exposure at runtime is a different layer of the stack. The tempo settles why that layer must be inline: Google Mandiant's M-Trends 2026 report put median attacker handoff at 22 seconds.

DeepInspect

Testing tells you where an LLM application fails. DeepInspect is the runtime layer that holds the line the tests keep finding.

DeepInspect is a stateless proxy on the AI request and response path. It does not depend on the model resisting an attack, which is the assumption every testing ceiling exposes. It binds each call to the caller identity your application supplies, classifies prompt and response content across PII, PHI, source code, and jurisdictional categories, authorizes each model call and tool invocation against role and content, and applies a deterministic pass, block, or modify decision inline and fail-closed, committing a per-decision audit record for every call. Keep the four testing categories to measure exposure and prevent regressions, and put an enforcement layer on the path to close what the tests find. Book a demo today.