DeepInspect vs Promptfoo: LLM Red-Teaming in CI and Runtime Enforcement

Promptfoo lives in the pull request. It is an open-source CLI and library where you declare tests and attacks in YAML, run them against your prompts and models, and wire the whole thing into CI so a red-team run gates a merge. Its red-teaming side ships dozens of attack plugins covering jailbreaks, prompt injection, SQL injection through an LLM, broken object-level authorization, and excessive agency, and it generates attacks tailored to the specific application under test. In 2025 OpenAI acquired it.

That developer-first design is why engineers reach for it. Fast feedback on every pull request, declarative configs a whole team can read, and attacks generated for your app rather than a generic list. It is genuinely strong at answering one question: can an attacker break this build.

Teams comparing Promptfoo to DeepInspect are usually holding a red-teaming tool next to a production control point. These sit at opposite ends of the deployment timeline. I want to make that explicit, because it is the whole distinction.

TL;DR

Promptfoo is an open-source, CI-native tool for evaluating and red-teaming LLM applications with YAML configs and dozens of attack plugins, now owned by OpenAI. DeepInspect sits inline in production between authenticated callers and model providers, decides whether each request proceeds based on identity and data, and commits a per-decision audit record. One attacks a build before it ships. The other governs traffic after it ships.

Promptfoo: declarative red-teaming in the pipeline

Promptfoo's model is the config file. You write a YAML spec describing the prompts, the providers, the test cases, and the red-team plugins to run, then execute it locally or in CI. The evaluation side compares outputs across models and prompt variants. The red-team side generates adversarial inputs from its plugin library and reports which attacks succeeded, from prompt injection and jailbreaks to authorization bypass and excessive-agency probes on agents.

Because it hooks into CI/CD, the results arrive as PR feedback. A new prompt template that regresses on an injection plugin fails the check before it merges, which keeps a class of vulnerability from reaching production. For an engineering team that already thinks in pull requests and pipelines, that fit is the selling point.

Promptfoo's position on the timeline is pre-deployment. It attacks a candidate build in a test setting and reports what got through, with no authority over live traffic. That is correct for a red-teaming tool and it is the boundary every such tool shares.

Attacking a build versus governing traffic

DeepInspect is a stateless proxy sitting between authenticated users or agents and any LLM endpoint in production. Every live call traverses it, and the decision happens before the provider sees the request.

Authority and timing separate them. Promptfoo tells you a build is susceptible to an attack class so you can fix it before shipping. DeepInspect governs the shipped system and decides, per request, whether this identity may proceed. A green Promptfoo run says the build resisted the generated attacks in CI last night. It has no bearing on the analyst who this afternoon sends regulated data through a call no plugin modeled. That runtime case belongs to an enforcement layer, and the case for inline enforcement is in why AI security must be inline.

Identity is the second gap. A Promptfoo test carries the attack it generated. It models no production caller, no role, no entitlement. DeepInspect treats that identity as the primary key of every decision and records it. The prompt-injection findings a tool like Promptfoo surfaces are the input to hardening; the runtime handling of injection is covered in how to prevent prompt injection.

Feature comparison

| Capability | Promptfoo | DeepInspect | |---|---|---| | Open source | Yes | No | | YAML-declared tests and attacks | Yes, core | No | | Attack plugin library | Yes, dozens | No | | CI/CD-native red-teaming | Yes, core | No | | Model and prompt comparison | Yes | No | | Runs in production on live traffic | No | Core | | Inline blocking of a request | No | Yes | | Identity-bound per-request policy | No | Core | | Data classification against custom classes | No | Yes | | Application-independent audit record | No | Core | | Primary buyer | Application and platform engineers | Security and compliance |

The overlap is narrow because the two live at different lifecycle stages. Promptfoo decides whether a build resists attack well enough to ship. DeepInspect governs what the shipped system is allowed to do.

Where Promptfoo is the better answer

If the question is whether this prompt template resists jailbreaks, or whether the agent exposes an authorization-bypass path under adversarial input, Promptfoo answers it and DeepInspect does not. CI-native red-teaming, declarative attack configs, and model comparison are its territory, and a policy proxy contributes nothing to pre-merge hardening.

Being open source and CI-native makes Promptfoo a strong fit for a developer workflow. Pre-deployment red-teaming is a real requirement, and Promptfoo serves it well. The OpenAI acquisition in 2025 is worth weighing if your team prefers an evaluation stack independent of a foundation-model vendor.

Where the gap opens

Two requirements move a team past what a red-teaming tool provides.

The first is control over live traffic. Production sees callers and prompts no attack plugin generated. A passing red-team run cannot refuse a request. An inline policy layer can, keyed to the caller's identity and the data in the prompt. The class of runtime risk a test cannot cover is the post-authentication gap.

The second is audit evidence. A red-team report is a pre-launch artifact. A regulator under the EU AI Act asks for a per-decision record of what was authorized in production, on storage the application cannot rewrite. The independence argument is in signed audit logs for AI requests.

Pick Promptfoo if

• Your team red-teams LLM applications as part of the development workflow.
• You want declarative YAML attack configs wired into CI/CD.
• Model and prompt comparison on every pull request is the immediate need.
• Open source and self-hosting are requirements rather than preferences.
• Nobody is asking you for per-decision production evidence.

Pick DeepInspect if

• You need live model calls refused, not builds attacked.
• An auditor or enterprise customer wants per-decision records tied to an identity.
• Policy must differ by role and be enforced on production traffic.
• You need an audit record on storage the application cannot alter.
• You operate under the EU AI Act, DORA, HIPAA, or a comparable mandate.

DeepInspect

This is exactly what DeepInspect does. DeepInspect sits inline between your users or agents and the LLM APIs they call. For every request and response, it evaluates identity, data classification, model authorization, and organizational policy, then makes a pass or block decision before the traffic reaches the model.

The record is the second half of the product. Every decision writes a structured entry containing the authenticated identity, the resolved role, the data classes found in the prompt, the policy version in force, and the outcome. That entry commits before the response returns, on storage the application cannot write to, which is what lets it serve as evidence rather than as a test report. Enforcement adds under 50 ms in internal testing against inference times of 500 ms to several seconds. A red-teaming tool and an enforcement layer belong in the same program, at different points in the lifecycle.

If you are facing the August deadline, let's talk.