Does the egress-control layer require us to MITM the AI provider TLS?

The layer terminates the application's TLS at the layer's hostname and opens a new TLS connection to the upstream provider with the deployer's provider credentials. The application sees the layer's certificate (issued by the deployer's internal CA or a publicly trusted certificate against the layer's hostname). The application does not need to trust the provider's certificate directly because the application is no longer talking to the provider directly. The pattern is the standard egress-proxy pattern with provider-side identity in the layer's connection to the upstream.

How does the layer handle the case where the application uses a long-lived API key against the provider?

The application sends the request to the layer with whatever credentials the layer requires for the application-to-layer authentication (an internal mTLS, an internal token, or a per-application short-lived credential). The layer authenticates the application, reads the propagated identity claims, evaluates the policy, and forwards to the upstream provider with the provider-side credentials the layer holds. The provider-side credential is rotated and managed in the layer's secret store, not in the application code.

Does the layer add observable latency the application has to tune for?

The layer adds typically under 50 ms of overhead per request in production. The application already absorbs 800 ms to 3 seconds of provider-side latency per turn. The layer's overhead is below the threshold a user notices against the existing latency budget. The layer can also short-circuit responses for cache-eligible requests (system prompts that repeat, response templates) which reduces total round-trip in many deployments.

Can the layer enforce policy on requests to a self-hosted LLM we run on our own infrastructure?

The layer sits in front of any HTTP-addressable LLM endpoint. A self-hosted Triton or vLLM deployment behind a load balancer is a valid upstream for the layer. The policy bundle, classification, and audit-commit mechanisms operate identically. The layer's adapter for the self-hosted endpoint speaks the endpoint's request shape (OpenAI-compatible API, Triton's HTTP API, or a custom shape) and normalizes the request to the layer's internal representation.

LLM Egress Control: The Per-Request Identity, Classification, and Audit Layer for AI Provider Traffic

LLM egress control is the request-time enforcement layer that sits between the corporate applications (and agents) and the external LLM endpoints they call. The layer's job is to read the identity the request carries, classify the prompt body the application sends, evaluate the policy bundle bound to the route, apply a pass, modify, redact, or block decision, and commit a per-decision audit record to a tamper-evident store. The layer is what distinguishes a regulated AI deployment from an unmanaged one. The unmanaged deployment lets every application reach api.openai.com or api.anthropic.com with whatever shared credential the team standardized on, and the resulting traffic is invisible to the audit and security functions. The egress-control layer makes the traffic visible at the request level and enforceable at the policy level.

I want to walk through the egress surface the layer covers, the four classes of data the layer reads at request time, the policy decisions the layer commits, the audit record format that satisfies EU AI Act Article 12, and the deployment topology that handles single-region and multi-region traffic.

The egress surface the layer covers

The AI request egress in a typical enterprise deployment reaches four endpoint families.

The first family is the direct provider endpoints: api.openai.com, api.anthropic.com, api.cohere.com, generativelanguage.googleapis.com, api.together.xyz, api.mistral.ai. The applications send requests directly to the provider's HTTPS endpoint.

The second family is the cloud-hosted provider endpoints: bedrock-runtime.<region>.amazonaws.com, <region>.api.cognitive.microsoft.com (Azure OpenAI), aiplatform.googleapis.com (Vertex AI). The applications send requests through the cloud provider's API surface.

The third family is the self-hosted LLM endpoints: a Triton or vLLM deployment in the deployer's own infrastructure, served behind a load balancer with TLS termination. The applications send requests to the internal endpoint the deployer owns.

The fourth family is the LLM-fronted SaaS the deployer subscribes to: a coding copilot's API, an analytics LLM tool, an embedded LLM in a vendor's product. The applications connect to the vendor's endpoint, which proxies the call to the underlying model provider.

The egress-control layer covers all four families. The layer sits between the calling application (or agent) and the upstream endpoint, terminates the TLS to the upstream endpoint, and applies the policy decisions on the request and response. The layer does not change what the application calls. It changes what the application is allowed to send and what the application receives.

The four classes of data the layer reads at request time

The first class is the identity context. The natural-person identifier from the propagated SSO, the agent identifier where applicable, the session identifier, and the route identifier. The identity context is carried in HTTP headers the application attaches.

The second class is the request body. The prompt, the system prompt, the model selection, the tool list, the function-calling schema, and any structured fields the caller supplies. A classifier passes over the request body and tags the data classes the prompt reaches.

The third class is the response body. The model's output, the tool calls the model emitted, the structured outputs, and the usage metadata. A classifier passes over the response and tags the data classes the model emitted.

The fourth class is the policy state. The policy bundle bound to the route, the policy version, and any rate-limit or quota counters the policy needs.

The policy decisions the layer commits

The layer commits five decisions per request. The first is identity verification: are the propagated identity claims present and valid for the route. The second is request-body classification: what data classes does the prompt reach. The third is request-body policy evaluation: does the policy bundle allow the classification for the identity and route. The fourth is response-body classification and redaction: what data classes does the response carry and which need to be masked before reaching the caller. The fifth is audit commit: the per-decision record commits to the tamper-evident store.

The decisions can be pass, modify, redact, or block. A pass leaves the request unchanged. A modify rewrites the request to fit the policy (typically by removing disallowed fields or stripping system-prompt overrides). A redact masks data above the caller's classification ceiling before forwarding or before returning. A block fails the request at the boundary with a structured error that the application surfaces to the user or the agent.

The audit record format that satisfies Article 12

EU AI Act Article 12 takes effect August 2, 2026 for high-risk systems and requires automatic event recording, identification of natural persons involved, and retention for at least six months. NIST AI RMF MANAGE 1.3 requires evidence that AI risks are tracked across the system lifecycle. ISO 42001 Annex A covers operational records.

The audit record format carries seven fields per request. The record carries the natural-person identifier the caller authenticated as, the agent identifier where applicable, the session and route identifiers, the policy version that evaluated the request, the decision outcome, the upstream model and version, and the integrity metadata that proves the record was not altered after the fact.

The record series joins on the natural-person and the agent identifiers. A regulator query for a data subject's interactions across a time window returns the request series ordered by time, with each record carrying the policy and decision. The retention window covers the regulatory floor (six months under Article 12, often multi-year under sector-specific obligations).

The deployment topology

The single-region topology places the egress-control layer in the same region as the applications that call it. The applications point at the layer's hostname instead of the upstream provider's hostname. The layer authenticates the upstream provider with the provider-side credentials the deployer holds. The applications carry the propagated identity claims as HTTP headers the layer reads.

The multi-region topology places one layer per region. Each region's applications point at the regional layer. The regional layer routes to the provider endpoint that satisfies the region's data-residency posture (EU users on EU-resident inference, US users on US-resident inference). The policy bundle per region describes the data-residency rules. The audit record commits to a region-resident store with cross-region replication for the deployer's global compliance function.

The hybrid topology runs an egress-control layer alongside a self-hosted LLM and the cloud provider endpoints. The same layer handles both surfaces with different upstream adapters. The policy bundle abstracts the provider difference. The audit record carries the upstream provider identifier on each record so the compliance function can review per-provider patterns.

DeepInspect

DeepInspect is the LLM egress-control layer for that topology. The product terminates the AI provider TLS, reads the request and response, verifies the propagated identity claims, evaluates the policy bundle per route, applies pass, modify, redact, or block decisions on both legs of the request, and commits per-decision audit records to a tamper-evident store with hash chaining across records.

The product runs as a stateless proxy. Round-trip overhead measures under 50 ms in production. The deployer's existing SSO propagates through. The policy bundles per route describe the classifications each route handles and the residency the route enforces. The record series satisfies AI Act Article 12 and supplies the per-decision detail GDPR Article 22 right-to-explanation requests now reach for.

If you are extending your egress program into the AI request path before August 2, let's talk today.