Crescendo Attack: Detect the Conversation, Not One Prompt

A Crescendo attack starts with a question the model can answer safely. The attacker then cites the model's reply, asks for a little more detail, and repeats the move until the conversation reaches material the model would have refused in one turn. The harmful intent lives in the trajectory. A filter that grades each new message as an isolated string sees a row of ordinary prompts and misses the staircase they form.

I want to focus on that staircase because it changes the unit a security team must test. One prompt is too small. The useful object is the conversation state that reached the model on the current HTTP request, along with the identity and route, plus the policy and prior decisions attached to it.

TL;DR

• A Crescendo attack escalates through benign-looking turns and reuses the model's own text to move toward a prohibited objective.
• Test complete conversations, including refusals and attacker backtracking, instead of scoring single prompts in isolation.
• An HTTP gateway can enforce policy on conversation history present in the routed request. Provider-held state and local agent execution remain outside that view.
• Record session correlation and current content, plus the destination and policy version. Record every permit or block decision so the full escalation can be reconstructed.

The Crescendo attack exploits conversational momentum

Mark Russinovich, Ahmed Salem, and Ronen Eldan introduced Crescendo as a multi-turn jailbreak that begins with a broad, innocuous question. Each later turn references earlier replies and raises the level of detail. Their USENIX Security 2025 paper reports evaluations against public systems including ChatGPT, Gemini, Llama, and Anthropic Chat. The paper's automated method, Crescendomation, outperformed the compared techniques on its AdvBench subset by 29% to 61% on GPT-4 and 49% to 71% on Gemini Pro.

The mechanism matters more than the leaderboard. The attacker gets the model to establish facts and vocabulary before settling into a cooperative pattern, then asks for the prohibited output. Recent context includes text the model generated itself, which the next request presents as settled conversation history. A direct request may trigger refusal while the accumulated path reaches the same objective in smaller steps.

This separates Crescendo from a fixed jailbreak phrase. Signature matching can catch a copied one-shot prompt after disclosure. Crescendo gives the attacker room to paraphrase and retreat after a refusal before resuming through a nearby topic.

Single-turn filters inspect the wrong unit

Picture ten request rows in a console. Nine are green and one is red. That view encourages the reviewer to ask which sentence crossed a threshold. Crescendo requires a second view: the ordered transcript, including which assistant statements the attacker quoted back and how the objective changed across turns.

A production detector needs access to the conversational material that actually reaches the model. For stateless chat APIs, applications often resend prior messages with each request. The assembled HTTP body then exposes the history to an inline inspection point. A provider-managed thread can work differently. If the application sends only a thread identifier and the latest message, the provider holds prior context behind its API, leaving the gateway with an incomplete transcript.

That distinction belongs in the architecture diagram. I would reject any control claim that says "conversation-aware" without naming where the conversation is reconstructed. The application can supply a normalized history or a trusted session summary. It can also supply risk signals computed upstream. The gateway can evaluate only what crosses its boundary.

The broader prompt injection techniques still matter, but Crescendo adds time and accumulated state to the attack surface.

Crescendo attack testing must preserve live history

Microsoft's current PyRIT Crescendo implementation makes the mechanics concrete. It generates progressively harmful prompts, checks the target response for refusal, backtracks when refusal occurs, scores progress toward the objective, and continues until it succeeds or reaches configured turn and backtrack limits. The implementation requires editable, multi-turn history because collapsing the exchange into one synthetic prompt changes the attack.

That gives a red team a useful test design. Preserve every attacker turn and model reply. Keep refused branches rather than deleting them from the evidence. Run the same objective through multiple conversation openings, because the setup language affects the path. Score the final outcome and the intermediate policy decisions separately.

An automated red teaming run proves that a model produced a result in an isolated test setup. Production validation needs a second replay through the deployed request path, with the real identity mapping, route restrictions, rate limits, and audit records active. The LLM security testing plan should therefore name two targets: the model endpoint and the complete production control path.

Session policy needs explicit evidence

A useful control record for a routed Crescendo test includes a stable session correlation value, the authenticated caller, model destination, current request classification, visible-history digest, policy version, decision, and timestamp. If the application supplies a conversation-risk score or normalized history, record that provenance as well. The record should say which content the decision point actually saw.

Rate and turn controls can narrow the attack window for a given identity and route. They remain coarse controls. A ten-turn ceiling limits one implementation of Crescendo while an attacker may compress the path or start fresh sessions. Content policy can block a current request when the assembled context exposes prohibited material. It cannot guarantee that a probabilistic model will maintain its refusal behavior after an allowed request reaches it.

That is why model guardrails remain one defense layer. Provider safety training can refuse the conversation. External policy can deny a routed request under a reproducible enterprise rule. Red-team evidence measures how the two behaved together without pretending they are the same control.

The HTTP boundary has a precise limit

DeepInspect's relevant boundary begins when an authenticated user or agent sends an HTTP request to an LLM through the proxy. It can inspect the request payload, evaluate application-supplied identity and context, apply per-role and per-route policy, and record the resulting permit, redact, block, or escalation decision.

For a chat request that carries full history, that payload can reveal the cumulative Crescendo path. For an API call that carries only the latest turn and a provider-side thread ID, the proxy sees the latest turn and identifier. It does not gain access to hidden provider state. The application must reconstruct the history or provide trusted session context if policy depends on earlier turns.

The boundary ends before several adjacent controls. DeepInspect cannot change the model's learned refusal behavior. It cannot inspect a conversation sent directly to a provider around the managed route. Local browser automation, stolen credentials, endpoint compromise, and tool execution outside routed LLM traffic need controls at those systems. A blocked model request prevents that request from reaching the model; it does not remediate an already compromised endpoint.

DeepInspect

DeepInspect sits inline on authenticated HTTP traffic deliberately routed between users or agents and LLM endpoints. When the request contains conversation history, it can evaluate that visible history with the supplied identity and active policy for the model route before forwarding the call. Each outcome produces a signed, tamper-evident decision record that can be correlated across the session.

Provider-side conversation state, model alignment, local execution, endpoint security, and bypass traffic remain outside that proxy boundary. For Crescendo testing, the practical requirement is explicit: send enough trusted conversation context through the controlled route for the policy decision you expect, then verify the resulting permit and block records against the red-team transcript.

Book a technical deep dive at deepinspect.ai.