Do PyRIT and garak replace the need for manual red teaming?

They replace the volume-scaling part of manual red teaming, not the judgment part. Automated tools run far more probe variations than a human tester can attempt in the same time, and they are good at surfacing known jailbreak patterns and encoding tricks at scale. They are weaker at novel attack creativity, business-logic-specific abuse cases that require domain knowledge of the specific application, and interpreting ambiguous results that need human judgment about severity. Most mature AI red teaming programs run both: automated tools for coverage and repeatability, manual testing for novel scenarios and business-context-specific abuse.

Does a passing red team result mean the deployment is safe?

A passing result against the specific probes run means those specific probes did not succeed, which is meaningfully narrower than "the deployment is safe." Automated red teaming tools test a library of known techniques and variations on them; a novel attack pattern not represented in that library will not be caught by definition. A passing result is evidence of resistance to known techniques, not a certification of overall safety, and should be read alongside the production enforcement evidence described above rather than as a standalone signal.

Can automated red teaming probes be run against agentic systems, not just chat models?

Yes, and this is where the field is expanding fastest. Testing an agent means probing not just what the model says but what actions it takes when given tool access, since an agent that can call APIs or execute code has a different and larger failure surface than a model that only generates text. PyRIT and similar frameworks have added support for multi-turn, tool-using scenarios, but the tooling here is younger and less standardized than text-only model probing, and the production-path gap described in this piece is larger for agentic systems, since there are more layers, tools, and permission boundaries between the model and the outcome.

Automated Red Teaming Finds the Jailbreak. It Does Not Confirm Your Gateway Stopped It.

Microsoft's PyRIT can run a multi-turn jailbreak strategy called Crescendo against a target model, escalating a conversation step by step until a model that would refuse a direct harmful request answers it anyway, and it can do this thousands of times across variations in the time a human red teamer would need for a handful of manual attempts. NVIDIA's garak runs a comparable library of probes covering prompt injection, data leakage, and toxic generation against any model reachable over an API. Both tools report the same kind of result at the end of a run: which probes made the model produce output it should not have. Neither tool tells you whether that same probe, sent by a real caller in production, would have reached the model at all.

TL;DR

Automated red teaming tools scale adversarial probing to thousands of attempts, finding model-level failures no manual review would catch in the same time.
The result is a model-behavior finding, measured against the model directly in a test environment.
That finding does not confirm what happens to the same probe in production, where policy, authentication, and traffic controls sit between the caller and the model.
Closing the loop requires replaying red-team findings against the actual production enforcement path, not just the model.

What automated red teaming actually measures

A red team run, automated or manual, answers a specific question: given this input, does the model produce an output that violates a safety or security expectation. PyRIT and garak automate the generation and iteration of those inputs, running variations, escalation strategies, and encoding tricks at a volume that makes the test statistically meaningful instead of anecdotal. That is real progress over the ad hoc manual jailbreak attempts that passed for AI red teaming as recently as 2023.

What the tools measure is the model's behavior when the probe reaches it directly, typically against a raw API endpoint in a test environment configured to match the model's deployment as closely as the tester can arrange. The result is accurate about the model. It says nothing about the production path the probe would have to survive in the actual deployment, because the test environment and the production stack are rarely the same system. Prompt injection benchmarking has the same limitation for the same reason: a benchmark score describes the model, not the deployment.

The gap between the lab and the deployment

Production AI deployments are not a bare model endpoint. Requests pass through authentication, often an application layer that shapes the prompt, sometimes a policy or filtering layer, before reaching the model, and the response passes back through some of the same layers before a user sees it. A red team run against the raw model endpoint tests none of that. It tests the model in isolation.

A model that fails a Crescendo probe in the lab might be protected in production by a policy layer that would have blocked the escalating conversation at request three, before the model ever saw the payload that triggered the failure. The reverse is also true and more dangerous: a model that passes every probe in the lab, because the lab environment enforces stricter input handling than production actually does, can be fully exposed once deployed, if production skips a control the test environment happened to include. Neither gap shows up in the red team report, because the report only describes what happened to the model.

Closing the loop requires replaying the finding, not just the probe

The fix is not a different red-teaming tool. It is running the same probes that failed against the model through the actual production request path, the one real traffic uses, and checking what happened there instead of assuming the lab result transfers. NIST's AI Risk Management Framework treats this kind of continuous measurement, checking a control's actual behavior rather than its intended behavior, as central to the Measure function, and that principle applies as much to red-team findings as to any other control.

A finding that "the model produced harmful output under probe X" becomes actionable only once it is paired with "and here is what happened when probe X was sent through the production path": blocked before the model, reached the model and got a redacted response, or reached the model and returned to the user unfiltered. Without that second half, a red team report is a list of model weaknesses with no evidence about which ones the deployment actually exposes, which is the same distinction between testing and continuous enforcement covered in building a repeatable injection testing program.

DeepInspect

This is where DeepInspect closes the loop rather than replacing the red team. DeepInspect sits inline on the actual production request path between authenticated users or agents and the LLM APIs they call, evaluating every request against identity, classification, and policy before it reaches the model. A red team probe replayed through that path, under a test identity and a real policy configuration, produces a genuine answer to the question a lab-only finding cannot: did the enforcement layer that real traffic passes through catch this.

Every evaluation writes a signed, per-decision record, so a security team validating a red team finding has an authoritative account of what the gateway did with that specific probe, not an assumption carried over from a test environment that may not match production. That record is also what turns a one-time red team exercise into a control a team can point to on demand, rather than a PDF from a testing cycle six months ago.

If your red team reports live in a different system than your production enforcement evidence, closing that gap is worth doing before the next test cycle. Let's talk today.

Continuous or near-continuous is the direction the field is moving, rather than the periodic point-in-time exercise that characterized early AI red teaming, and it converges with the four categories a mature testing program tracks whether the tests originate from a red team exercise or an internal QA pass. Models get updated by providers on schedules the deploying organization does not control, prompts and system instructions change as the application evolves, and a probe that failed six months ago may behave differently against a model version that shipped last week. Running automated probes on a recurring schedule, and specifically after any model version change or system prompt update, catches regressions a one-time assessment misses entirely.