What is the most common vendor-evaluation mistake?

Selecting based on a feature checklist instead of the architectural criteria. A feature checklist asks "does the vendor support X" and the vendor answers yes. The architectural criteria ask "where does X happen in the request path, what does X read, and what does X write to the audit record." The architectural questions distinguish vendors whose products share a feature name but cover different inspection targets at different layers of the stack.

How does the EU AI Act August 2 deadline affect the evaluation timeline?

The high-risk system requirements of the EU AI Act take effect on August 2, 2026. Deployments classified as high-risk have to demonstrate the records the regulator expects from that date forward. A full procurement cycle (questionnaire, proof of concept, legal and commercial) runs four to six weeks. A deployment that starts the evaluation in mid-June has roughly six weeks to complete and deploy before the deadline. Deployments that started later have to compress the cycle and accept higher residual risk.

Should the buyer evaluate AI security vendors and AI gateway vendors in the same matrix?

The two products solve different problems and the matrix should distinguish them. An AI gateway routes requests and handles fallback, rate limiting, and observability across upstream models. An AI security vendor runs identity-bound policy and commits the audit record series the regulator consumes. The deployment runs both. The matrices share some questions (model coverage, integration mechanics) and diverge on others (the audit record format and the policy versioning surface).

How does the buyer test fail-closed behavior in the proof-of-concept phase?

The proof of concept simulates the failure of the audit storage by disconnecting the inspection layer from the storage substrate. The vendor's product has to surface 5xx errors to the application and refuse to forward requests until the storage substrate is reachable again. The proof of concept also simulates the policy store failure by disconnecting the inspection layer from the policy administration point. The product has to refuse to evaluate against an unknown policy version and surface the failure to the operator. A product that succeeds under either condition does not satisfy the fail-closed criterion the regulator expects.

What artifacts should the buyer collect for the audit trail of the vendor evaluation?

The buyer collects the vendor questionnaire responses, the proof-of-concept results (recorded audit records, policy version evidence, fail-closed test results), the SOC 2 Type II report, the data processing terms, the Business Associate Agreement where applicable, the architectural diagram the vendor provides, and the security team's written sign-off on the architectural criteria. The artifacts compose the audit trail the buyer's audit reviewer reads when the reviewer asks how the vendor was selected. The artifacts also compose the renewal review evidence one year later.

How to Evaluate AI Security Vendors: The 12 Questions a Production Buyer Asks Before Signing

AI security vendor evaluation reaches the buyer's desk in three different shapes. A regulatory deadline (EU AI Act August 2, DORA in effect, Fannie Mae LL-2026-04 on August 6) creates pressure to select a vendor within weeks. An incident inside the deployment (a shadow AI data leak, a prompt injection event, a regulator inquiry) creates pressure to add a control immediately. A platform consolidation effort folds an AI security vendor decision into a broader rationalization project. The pressure varies and the question the buyer answers stays the same: which vendor's architecture and operational behavior produces the audit record series and the enforcement posture the security team and the audit reviewer accept.

I want to walk through the twelve questions the production buyer asks each vendor, the answer pattern that satisfies the regulator and the security team, and the way the resulting matrix gets used inside the procurement cycle.

The first four: inspection boundary and what the vendor reads

The first question is where the inspection layer sits in the request path. The answer the buyer wants to hear is "at the HTTP boundary between the calling identity and the LLM endpoint, terminating the upstream TLS." A vendor whose product sits at the network layer reads TCP and TLS metadata and cannot see the prompt content. A vendor whose product runs inside the inference path reads inside the application boundary and fails the regulator's write-path independence test.

The second question is what the inspection layer reads at decision time. The answer the buyer wants is identity context, route context, request content, data classification output, and policy state. A vendor whose product reads only the network metadata cannot supply the input data fields the regulator expects. A vendor whose product reads only the response cannot evaluate the prompt at decision time.

The third question is whether the inspection layer covers HTTP-based LLM endpoints across providers. The answer the buyer wants is "any HTTP-based LLM endpoint." A vendor whose product covers only one cloud (AWS Bedrock, Azure OpenAI) cannot cover the deployment's full footprint when the deployment runs OpenAI, Anthropic, Bedrock, and an on-prem inference endpoint side by side.

The fourth question is what the inspection layer commits at decision time. The answer the buyer wants is a per-decision audit record with identity, route, data classification, policy version, decision outcome, model and version, and integrity metadata. A vendor whose product writes records into the application's existing logs produces records inside the application boundary, which fails the independence test the regulator applies.

The next four: policy management and audit record format

The fifth question is how policies are managed and versioned. The answer the buyer wants is a policy administration point that manages versioned policy bundles, supports rollback, and exposes the policy version in the audit record at decision time. A vendor whose product manages policies through configuration files the operator edits in place produces audit records without a stable version identifier.

The sixth question is what the audit record schema covers. The answer the buyer wants is a documented, stable schema with the seven fields above. The schema has to be the same across model providers so the deployment's downstream consumers (SIEMs, GRC archives) read records with a fixed parser. A vendor whose schema varies by model or whose record format is free-form text produces records the downstream consumers cannot parse uniformly.

The seventh question is how the audit record series is integrity-stamped. The answer the buyer wants is per-record cryptographic signatures and a hash chain that links each record to the previous one. The reviewer verifies the chain at read time. A vendor whose records are plain-text rows in a database without signatures cannot prove integrity at read time.

The eighth question is where the audit record series is stored. The answer the buyer wants is durable, append-only storage with retention controls that match the regulatory regime. EU AI Act Article 12 expects retention for the lifetime of the system. DORA Article 19 expects retention aligned with the operational risk framework. Fannie Mae LL-2026-04 expects retention aligned with the lender's existing record retention. A vendor whose product writes records only to the SIEM (which has its own retention) shifts the retention obligation to the SIEM operator.

The next two: regulatory mapping and operational behavior

The ninth question is which regulatory regimes the vendor maps the audit record against. The answer the buyer wants is a documented mapping from the record fields to EU AI Act Article 12, DORA Article 19, Fannie Mae LL-2026-04, NIST AI RMF, HIPAA 45 CFR 164.312, ISO 42001 Annex A controls, and the sector-specific regimes the deployment operates under. The mapping is the evidence the compliance team uses when the auditor asks how the record satisfies each obligation.

The tenth question is what the inspection layer does under blocking-dependency failure. The answer the buyer wants is fail-closed: a request whose audit record cannot be committed cannot proceed, a request whose policy bundle cannot be loaded cannot proceed. The application sees a 5xx, retries, and either succeeds or surfaces the failure to the caller. A vendor whose product fails open under either condition produces decisions that did not get recorded or evaluated.

The last two: integration and procurement mechanics

The eleventh question is how the inspection layer integrates with the existing AI traffic topology. A deployment that runs an AI gateway like LiteLLM, Portkey, or Helicone for routing has the inspection layer sit in front of the gateway or alongside it. A deployment that calls AI providers directly has the inspection layer as the egress proxy the applications target. The answer the buyer wants is the integration model that fits the deployment's current topology and that the operator can roll forward and back in stages.

The twelfth question is how the procurement and security review cycles handle the vendor. The answer the buyer wants covers the vendor's SOC 2 Type II report, the vendor's penetration test history, the vendor's data residency and processing boundaries (especially for healthcare and finance), the Business Associate Agreement availability (for HIPAA-regulated deployments), and the vendor's incident response process. A vendor who cannot produce these artifacts at the table fails the procurement gate before the technical evaluation matters.

The answer pattern that satisfies the buyer

The vendor whose answers across the twelve questions converge on the same architectural pattern is the vendor the buyer selects. The pattern is an HTTP-boundary inspection layer that reads request and response in cleartext, evaluates identity-bound policy against versioned bundles, applies pass, block, or modify, commits per-decision audit records with the seven fields and cryptographic integrity to durable storage, covers any HTTP-based LLM endpoint, fails closed under blocking-dependency failure, maps the records to the regulatory regimes the deployment operates under, and integrates with the existing AI traffic topology.

A vendor whose answers cover ten of the twelve produces a partial answer the audit reviewer detects on the first read of the record series. A vendor whose answers cover eight or fewer is selling a different product than the regulated deployment needs.

How the matrix gets used in the procurement cycle

The matrix runs through three phases the procurement cycle goes through. The first phase is the questionnaire phase. The buyer sends the twelve questions to each vendor in the shortlist. The vendor returns written answers within five business days. The buyer scores the answers against the answer pattern above.

The second phase is the proof-of-concept phase. The buyer asks the top two or three vendors to demonstrate the inspection layer against a synthetic deployment that mirrors the buyer's production topology. The demonstrations cover the audit record format on real requests, the policy versioning behavior on a rolled-forward policy, and the fail-closed behavior under simulated audit-storage outage. The buyer's security team reviews the records and the operational behavior.

The third phase is the procurement and legal phase. The buyer's legal and procurement teams review the vendor's SOC 2 Type II, the data processing terms, the Business Associate Agreement (where applicable), and the commercial terms. The technical evaluation and the procurement evaluation converge on the same vendor or the buyer goes back to the proof-of-concept phase with a different shortlist.

The full cycle runs four to six weeks under normal conditions. Deployments under the EU AI Act August 2 deadline that have not started the evaluation by mid-June are operating outside the timeline the deadline supports.

DeepInspect

This is the architectural pattern DeepInspect was built to fit. DeepInspect is the HTTP-boundary inspection layer between the calling identity and any LLM. The inspection layer reads request and response in cleartext, evaluates identity-bound policy against versioned bundles, applies pass, block, or modify, and commits per-decision audit records to durable, append-only storage with cryptographic integrity before the response forwards. The inspection layer covers any HTTP-based LLM endpoint and fails closed when its blocking dependencies are unavailable.

The audit record series carries identity, route, policy version, data classification outcome, decision outcome, model and version, and integrity metadata in a format that EU AI Act Article 12, DORA Article 19, Fannie Mae LL-2026-04, NIST AI RMF, HIPAA 45 CFR 164.312, and ISO 42001 reviewers consume. End-to-end inspection-layer overhead measures under 50 ms in production. The deployment integrates with existing AI gateways (LiteLLM, Portkey, Helicone) and with direct-call AI traffic.

If you are running an AI security vendor evaluation ahead of the EU AI Act August 2 deadline, book a technical deep dive at deepinspect.ai.