AI Gateway Architecture: The Components That Sit Between an Enterprise Caller and an LLM Endpoint

An AI gateway is the HTTP proxy that sits between an enterprise caller and an LLM endpoint. The architecture has six core components, and each component is a placement decision that ties to a regulatory obligation, an operational property, or an integration with the existing enterprise stack. The components are TLS termination, identity binding, request inspection, policy evaluation, the model router, and the audit record emitter.

I want to walk through each component, what it does, where it sits relative to the others, and the operational consequences of getting the placement right or wrong.

TL;DR

An LLM gateway architecture has six components: TLS termination, identity binding, request inspection, policy evaluation, a model router, and an audit record emitter. Each one is a placement decision tied to a regulatory obligation, an operational property, or an existing piece of the enterprise stack, from the corporate IdP to the SIEM. In production the gateway runs as a cluster behind a load balancer with an explicit fail-open or fail-closed decision for outages, and the classifier inside request inspection is usually the first component to need capacity planning under real load.

Component one: TLS termination

The gateway terminates TLS at the inspection point. The TLS termination is what lets the gateway read the request body, which carries the prompt content. Without TLS termination at the gateway, the prompt content remains encrypted to the model provider's endpoint and the inspection cannot read it.

The TLS termination raises a certificate question. The client (the application or the user's browser) needs to trust the gateway's certificate for the LLM endpoint domain. Programs typically handle this with a corporate root certificate the device fleet already trusts, with the gateway issuing leaf certificates for each LLM endpoint the program routes through. The certificate management belongs in the PKI infrastructure the program already operates.

The path breaks into two independent TLS sessions with the gateway sitting in the middle:

• Client to gateway: TLS terminates at the gateway using the gateway's own certificate, which the client's PKI already trusts.
• Gateway to LLM endpoint: the gateway opens a new TLS session and verifies the provider's certificate itself.

Component two: identity binding

The identity binding step authenticates the caller against the corporate identity provider. It extracts the IdP token from the request (typically a JWT in the Authorization header), verifies the signature against the IdP's public key, and attaches the verified identity to the request context. That identity context flows downstream through the policy evaluation and the audit record emission.

For agentic traffic, the identity binding includes the delegation chain: the agent's service identity plus the originating user identity from the delegation token. The chain carries through the request context for the policy and the record.

The identity binding is the field the EU AI Act Article 19 record references. The placement at the gateway is what lets the identity show up on every record without each application team wiring it through. I go deeper on the identity model itself, including how the delegation chain is verified and how it maps to policy scope, in identity-aware AI gateway architecture.

Component three: request inspection

The request inspection parses the request body, identifies the prompt fields per the LLM API contract (messages for OpenAI Chat Completions and Anthropic Messages, contents for Google Generative AI, inputText for AWS Bedrock Runtime), and runs the classifier against the prompt content.

The classifier output: category labels (PII, PHI, source code, customer data, organization-defined categories), confidence scores per label, and the span (start offset, length) of each labeled section. The inspection produces a structured classification result the policy can match against.

The inspection runs on the request body and, on the response side, on the model response body. The two inspections share the classifier and the policy surface but emit separate decisions for the request side and the response side.

Component four: policy evaluation

The policy evaluation receives the identity context and the classification result and returns a decision. It runs as a function over the (data, identity, model) triple. The decision is permit, redact, or block; some gateways add escalate for the human-review pattern.

The policy state is versioned, and each evaluation records the policy version that decided. The version is what lets the program reconstruct the decision basis when the policy changes between the original decision and the audit review.

Here is what two rules can look like inside one policy version:

• block_phi_for_non_clinical: when the classification is PHI and the identity's role is not clinical or research, the decision is block.
• redact_pii_for_external_models: when the classification is PII and the model tier is external, the decision is redact.

Component five: model router

The model router decides which LLM endpoint receives the forwarded request. The decision can route based on the model the caller requested, the policy tier the caller's identity holds, the model availability state, or the regional jurisdiction the data has to stay inside.

The router is the integration point for multi-provider deployments, and the same policy and record surface covers every destination underneath it:

• OpenAI: api.openai.com
• Anthropic: api.anthropic.com
• Bedrock: bedrock-runtime.us-east-1.amazonaws.com
• Self-hosted: an internal inference endpoint the program operates

The router is also the integration point for data residency. A policy can require certain data tenants' requests to route to in-region endpoints, with the routing decision recorded on the audit series. The router is where cost-based model routing plugs in too, since the same decision point that enforces jurisdiction can also send a batch workload to a cheaper model.

Component six: audit record emitter

The audit record emitter commits the per-decision record before the response returns to the caller. It carries the timestamp, identity, classification, decision, policy version, request and response hashes, and an integrity signature chained from the previous record. The series is tamper-evident and queryable for audit sampling.

The emitter writes to a record store (a tamper-evident log, often backed by a database with append-only constraints or a write-once storage tier). From there, records flow into the SIEM (Splunk, Elastic, Datadog) through standard log forwarding. The compliance team queries the SIEM or the record store directly for audit sampling.

How the components interact on a single request

1. Client sends the request to the gateway with an IdP token in Authorization.
2. Gateway terminates TLS and reads the request body.
3. Identity binding verifies the IdP token and attaches identity context.
4. Request inspection parses the body, runs the classifier, and produces labels.
5. Policy evaluation weighs data, identity, and model, and returns permit, redact, or block.
6. A block decision returns an error to the client with no model call; a redact decision rewrites the prompt with typed placeholders first.
7. A permit or redact decision sends the request through the model router to the LLM endpoint.
8. The response comes back and passes through response inspection, which produces a second policy decision.
9. The audit record commits the full request, response, and both decisions, then the response returns to the client.

Integration with the existing enterprise stack

The gateway integrates with three existing systems: the corporate IdP supplies the identity, the SIEM receives the audit records, and the PKI fleet trusts the gateway's certificates. The integration patterns are standard: OIDC or SAML for the IdP, syslog or HEC for the SIEM, corporate root certificate for the PKI.

For programs running the gateway in front of multiple LLM providers, the deployment is typically as a sidecar or a cluster behind a load balancer the application tier already uses. The latency overhead measures under 50 ms end-to-end in internal testing, which sits inside the LLM inference variance.

Regulatory framing

The gateway architecture maps to specific regulatory obligations. EU AI Act Article 12 and Article 19 on record-keeping land on the audit record emitter. Article 26 on deployer obligations reaches the policy evaluation step and the human-oversight integration. HIPAA Security Rule 45 CFR 164.312(b) on audit controls ties to the record emitter and the identity binding. PCI DSS version 4 requirement 10 on logging pulls from that same record series.

NIST AI RMF MEASURE function lands on the classification and decision metrics the gateway emits. MANAGE function lands on the feedback loop between the records and the policy refinement.

Running the gateway in production

An architecture diagram is not a production deployment, and the gap between the two is where most LLM gateway rollouts stall. Three things change once real traffic hits the gateway.

High availability comes first, because the gateway sits directly in the request path, so a single instance is a single point of failure for every LLM call the enterprise makes. Production deployments run the gateway as a cluster behind a load balancer, matching the pattern already used for Azure OpenAI gateway deployments, with health checks pulling a bad instance out of rotation before it drops traffic.

Failure mode comes second. When the gateway itself is unreachable, the policy decision is fail-open or fail-closed, and that decision has to be made explicitly rather than left to whatever the load balancer does by default. Regulated workloads (PHI, financial data) fail closed: no gateway, no model call. Lower-sensitivity workloads sometimes fail open with a logged exception, trading availability for a gap in the record series during the outage window.

Scaling comes third, and the classifier step in request inspection is the component most likely to become the bottleneck under load, since it runs synchronously on every request and response. Production deployments benchmark the classifier's p99 latency at expected peak volume before launch, not after the first incident.

DeepInspect

DeepInspect ships the AI gateway architecture with all six components in one integrated proxy. TLS termination at the edge, IdP integration for identity binding (Okta, Entra ID, Ping, OIDC), deterministic classification on PII, PHI, source code, customer data, and organization-defined categories, policy evaluation against the (data, identity, model) triple, multi-provider routing, and a tamper-evident audit record series the SIEM consumes through standard log forwarding.

For programs designing an AI gateway from the architecture up, the six-component model is the reference shape. The integrated deployment is what makes the gateway operational without per-team integration work.

Book a demo today.