How is agent red teaming different from LLM red teaming?

LLM red teaming targets the model, testing whether prompts produce content it should refuse. Agent red teaming targets the deployed system, testing whether a chain of interactions produces actions the operator never authorised. The second usually succeeds without any model-level jailbreak, because the failure lives in composition rather than in the model.

How long should an exercise run?

Long enough for multi-turn effects to appear, which in practice means sessions of ten to twenty turns and a campaign spanning days rather than hours if memory persistence is in scope. Short single-turn testing systematically misses drift, chaining, and memory poisoning.

What should the deliverable contain?

Reproducible chains rather than a list of prompts, each with the sequence, the tools invoked, the identity used, and the end state reached. Each chain should carry a proposed control from outside the agent. A deliverable that recommends only prompt changes has documented the problem without addressing it.

Can automated tools replace a manual exercise?

They cover breadth well, generating large volumes of adversarial inputs and regression-testing known chains cheaply. Novel composition attacks, the ones combining tools in ways nobody designed for, still come from a human who understands the business context. Most programmes use automation for regression and humans for discovery.

Does red teaming satisfy a regulatory requirement?

It supports one rather than satisfying it. Frameworks including the EU AI Act and NIST AI RMF expect adversarial testing as part of risk management, alongside enforced controls and records demonstrating those controls operated. Testing evidence without enforcement evidence answers half the question.

AI Agent Red Teaming: Attacking the Loop, the Tools, and the Delegated Authority

Red teaming a chatbot asks whether the model can be talked into saying something it should refuse. The output is text, the blast radius is a screenshot, and a refusal rate is a reasonable score.

Agents change that question entirely. An agent holds tools, carries memory across turns, and acts under authority delegated from a human who is no longer watching. The interesting failure is the agent doing something rather than saying something, and it usually arrives through a chain where no single step looks alarming.

I want to lay out the attack classes that are specific to agents, walk a multi-turn escalation end to end, and cover the part most exercises handle badly: converting findings into controls that hold.

What changes when the target is an agent

Four properties of agent architecture create attack surface that single-turn testing never touches.

The loop means an attacker gets many turns to steer, and each turn's output becomes the next turn's input. Tools mean model output crosses into execution: a database query, an API call, a file write. Memory means an injected instruction can persist past the conversation that delivered it. Delegated authority means the agent carries permissions a human granted once, and those permissions apply to every subsequent step, including steps the human never anticipated.

Testing a prompt in isolation exercises none of these.

Six agent-specific attack classes

1. Indirect prompt injection through tool output

The agent retrieves a document, a web page, or a support ticket, and that content contains instructions. Because retrieved content and user instructions land in the same context window with no privilege boundary between them, the model has no reliable basis for treating one as data and the other as command.

The test: plant instructions in every source the agent reads. A calendar invite description, a PDF in the knowledge base, a row in a CRM record, an HTTP response body. Then see how far the planted instruction travels.

2. Goal drift across turns

Rather than asking for the prohibited thing, an attacker moves the agent's working objective one acceptable step at a time. The first turn establishes a benign task. Turn four reframes the scope, and by turn nine the attacker requests something the agent would have refused at turn one, now consistent with the objective it believes it holds.

Scoring here is different from a refusal rate. What matters is how many turns the drift needed and whether any single turn would have tripped a reviewer.

3. Tool chaining to an unintended capability

Individually safe tools compose into something neither was scoped for. A read-only database tool plus an email tool equals exfiltration. A file-read tool plus a web-fetch tool equals the same. The agent was never granted an exfiltration capability, and it assembled one.

The test: enumerate the tool set, then map every pair and triple for reachable end states. This is tedious and it finds real issues.

4. Memory poisoning

An instruction written into persistent memory or a summarisation buffer survives the session that planted it. Later sessions, possibly under a different user, inherit it. The attack decouples the moment of injection from the moment of effect, which defeats testing that treats each conversation as independent. Related mechanics in AI agent context window poisoning.

5. Delegated authority escalation

The agent authenticates to downstream systems with a static service credential, which is how most agents work today. That credential grants standing access for any prompt and any data context, so the practical authority of the agent is the union of everything the credential can reach rather than the narrow scope the user intended.

The test: establish what the agent's credential can reach, then construct a request that stays inside the credential's permissions and outside the user's intent.

6. Multi-agent trust abuse

In a crew or handoff architecture, agents accept each other's output as trusted input. Compromise the weakest agent, usually the one with the broadest input surface, and its output becomes an injection vector into agents that never touched hostile input directly.

A worked escalation

The chain below is the shape these findings usually take. Each step is individually defensible and the sequence is not.

Nothing in that chain required a model jailbreak. The agent followed instructions from a source it had no mechanism to distrust, then used tools it was legitimately granted. A refusal-rate metric scores this run as a clean pass.

Scoping the exercise

Three scoping decisions determine whether the exercise produces anything useful.

Test the deployed system rather than the model. The interesting failures live in the composition of retrieval, tools, memory, and permissions, and a model-only test finds none of them. Model-level work belongs in a separate track, covered in AI red team methodology.

Run multi-turn by default, with sessions long enough for drift and memory effects to appear. Ten to twenty turns is where the useful findings start.

Include the identity dimension explicitly. Run the same attack chain as different roles, and check whether the agent's effective authority tracks the requesting user's entitlements or the service credential's. In most deployments it tracks the credential, and that finding alone justifies the exercise.

Every action the agent takes needs recording during the test, including the ones that succeeded. A red team without a per-step record of which identity invoked which tool against which data cannot reconstruct why a chain worked.

Turning findings into controls

This is where most exercises lose their value. A report describing twelve successful chains produces a prompt-hardening sprint, and prompt hardening degrades under adversarial pressure. Stanford Trustworthy AI research and the AIUC-1 Consortium briefing found model-level refusal behaviours significantly degraded under targeted fine-tuning and adversarial pressure (Help Net Security).

Findings hold when they convert into enforceable controls, and the mapping is fairly mechanical:

| Finding | Durable control | |---|---| | Indirect injection via retrieved content | Classify retrieved content and enforce policy on the resulting model call | | Goal drift over turns | Evaluate each call independently rather than trusting session state | | Tool chaining to exfiltration | Constrain the reachable capability set per delegated scope | | Memory poisoning | Treat memory as untrusted input on read, classify before use | | Delegated authority escalation | Per-call authorization keyed on the requesting identity, not the credential | | Multi-agent trust abuse | Independent policy evaluation at each agent boundary |

The pattern across the right column is that the control sits outside the agent and evaluates per call. A control living inside the agent's prompt is subject to the same manipulation the exercise just demonstrated.

NIST's AI agent identity and authorization work frames this as three pillars: agent identity, delegated authority, and action lineage (NIST NCCoE). Establishing identity falls to the application. Delegated authority and action lineage need an enforcement layer at the AI call boundary, separate from the agent and separate from the model.

Re-testing

Red teaming is worth little as a one-off. Agents change when a tool is added, a prompt is edited, a model version ships, or a framework updates, and any of those can reopen a closed finding. The exercise belongs on a cadence tied to those change events, with the successful chains from the last round kept as a regression suite. Detail in AI red teaming workflow.

DeepInspect

This is the problem DeepInspect was built to solve at the layer it owns. DeepInspect is a stateless proxy on the HTTP path between authenticated users or agents and any LLM endpoint, evaluating identity, data classification, model authorization, and organizational policy on every call.

For agent traffic the evaluation runs per call rather than per session, so an agent that drifts across turns or inherits an injected instruction still meets a fresh policy decision at each step, keyed on the requesting identity rather than on a standing service credential. Every decision writes a structured record showing which agent, acting for which principal, sent what data class to which model with what outcome, committed before the response returns on storage the agent runtime cannot modify. That record is also what makes a red team exercise reconstructable afterwards. Two limits are worth stating plainly: tool calls that execute locally inside the agent process sit outside an HTTP gateway's boundary, and so does anything an agent does that never touches a model endpoint. Book a technical deep dive at deepinspect.ai.