LLM Gateway vs API Gateway: Where the Inspection Targets Diverge and Why You Need Both
API gateways inspect HTTP requests against rate limits, authentication tokens, and schema validation. LLM gateways inspect the prompt body, the response body, the identity carrying the request, and the policy bundle bound to the AI route. The inspection targets differ. The two run side by side in a production deployment. This piece walks through the inspection targets each gateway covers, the decisions each commits at request time, the audit record each produces, and the topology where the two compose.
An API gateway is the inspection layer between client applications and the HTTP services the applications consume. The gateway terminates the client TLS, authenticates the request against an API key or a signed token, applies rate limits and quota, validates the request schema against an OpenAPI specification, and forwards to the upstream service. The pattern is mature: Kong, Apigee, AWS API Gateway, Tyk, and Envoy all run this shape in production. An LLM gateway is the inspection layer between authenticated callers and the LLM endpoints the callers reach. The gateway terminates the AI provider TLS, evaluates identity-bound policy against the prompt body, applies a pass, modify, redact, or block decision, commits a per-decision audit record, and forwards to the upstream model. The two gateways look similar at the level of "an HTTP middlebox" and operate on different inspection targets.
I want to walk through what each gateway actually inspects, the decisions each commits at request time, the audit record each produces, and the topology where the two compose in a production AI stack.
What the API gateway inspects
The API gateway reads four classes of data on each request.
The first class is the route metadata. The path, the HTTP method, the host header, and the query parameters. The gateway matches the request against a route table and applies the route's policy.
The second class is the authentication token. An API key in a header, a JWT in the Authorization header, a signed AWS sigv4 request, or an mTLS client certificate. The gateway verifies the token, resolves the caller, and attaches the caller's identity to the request context.
The third class is the schema. The gateway validates the request body and headers against an OpenAPI or Protocol Buffers schema. A request that does not match the schema fails at the gateway with a structured error.
The fourth class is the rate limit and quota state. The gateway reads the caller's quota counters and applies the rate-limit policy. A caller that exceeds the quota receives a 429.
The API gateway does not read the prompt content the caller sends to an LLM endpoint. The gateway sees the AI request as an HTTP POST against the model provider's domain, validates the schema (often only at the envelope level), and forwards. The body's semantic content is invisible to the gateway.
What the LLM gateway inspects
The LLM gateway reads four classes of data on each request.
The first class is the identity context. The natural-person identifier from the propagated SSO, the agent identifier if the caller is an autonomous agent, the session identifier, and the route identifier. The gateway evaluates the identity against the policy bundle bound to the route.
The second class is the prompt body. The gateway reads the prompt content, the system prompt, the model selection, the tool list, the function-calling schema, and any structured fields the caller supplies. A classifier passes over the prompt content and tags the data classes the prompt reaches.
The third class is the response body. The gateway reads the model's output (text, tool calls, structured outputs) and the usage metadata. A classifier passes over the response and tags the data classes the model emitted.
The fourth class is the policy state. The gateway evaluates the policy bundle bound to the route, applies the decision (allow, modify, redact, or block), and commits the per-decision audit record.
The LLM gateway does not handle the rate-limit and quota state for the application's non-AI traffic. The gateway also does not handle the schema validation for the application's REST API surfaces. Those obligations remain with the API gateway.
The decisions each gateway commits at request time
The API gateway commits four decisions per request. Authentication: is the caller's token valid. Authorization: is the route the caller is targeting allowed for the caller. Schema validation: does the request body match the expected shape. Rate limit: is the caller within quota. Each decision is a yes/no with a structured error on no.
The LLM gateway commits five decisions per request. Identity verification: does the propagated identity satisfy the route's policy. Classification: what data classes does the prompt reach. Policy evaluation: what does the policy bundle say about the caller, classification, and route. Decision application: pass, modify, redact, or block. Audit commit: what record carries the decision into the tamper-evident store.
The two decision sets are non-overlapping. A request can pass the API gateway (valid token, allowed route, valid schema, under quota) and fail the LLM gateway (caller authorized for the route but the prompt reaches a classification the policy blocks). The opposite also holds: a request can pass the LLM gateway and still fail at the API gateway if the surrounding HTTP envelope is malformed.
The audit record each produces
The API gateway emits an access log per request. The standard fields are the timestamp, the caller's resolved identity, the path, the method, the response code, the response latency, and the bytes transferred. The format is widely standardized (Common Log Format, JSON access logs, OpenTelemetry traces). The consumer of the log is the application observability stack.
The LLM gateway emits a per-decision audit record per request. The fields are the natural-person identifier, the agent identifier, the session and route identifiers, the policy version that evaluated the request, the decision outcome, the upstream model and version, and the integrity metadata that proves the record was not altered after the fact. The format is purpose-built for the EU AI Act Article 12, NIST AI RMF MANAGE 1.3, and ISO 42001 record obligations. The consumer of the record is the regulator, the auditor, and the deployer's compliance function.
The two record streams are stored separately. The API access log goes to the observability stack (Elasticsearch, ClickHouse, Splunk). The LLM audit record goes to the tamper-evident store with hash chaining across records. Combining them into a single index loses the integrity properties the audit store provides.
The topology where the two compose
The two gateways run in series for AI traffic. The API gateway handles the application-level envelope (auth, schema, quota). The LLM gateway handles the AI-specific request semantics (identity-bound policy, classification, per-decision audit). The application sees a unified inspection layer. Each gateway operates on its specialized inspection target.
The deployer's existing API gateway investment continues to handle the non-AI surface. The LLM gateway extends the coverage to the AI surface. The audit posture the AI Act expects is satisfied by the LLM gateway's record stream, not by the API gateway's access log.
DeepInspect
DeepInspect is the LLM gateway for that topology. The product terminates the AI provider TLS, reads the request and response, verifies the propagated identity claims, evaluates the policy bundle per route, applies pass, modify, redact, or block decisions, and commits per-decision audit records to a tamper-evident store with hash chaining across records.
The product runs as a stateless proxy in front of the upstream model endpoints. It composes with the deployer's existing API gateway (Kong, Apigee, AWS API Gateway, Envoy, Tyk) by sitting on the AI-bound leg of the routing fork. The API gateway continues to handle the non-AI surface, the application's authentication, and the rate-limit budget. The LLM gateway handles the AI request semantics and the audit record series.
If you are extending your API gateway program into the AI request path and you owe an Article 12 audit, book a technical deep dive at deepinspect.ai.
Frequently asked questions
- Can we use our existing API gateway as the LLM gateway?
The existing API gateway can handle the authentication, schema validation, and rate-limit obligations for AI-bound traffic. The gateway does not classify the prompt body, evaluate identity-bound policy against the classification, or commit the per-decision audit record. The AI Act Article 12 obligation requires the per-decision record. The deployer that reuses the API gateway for the AI surface satisfies the application-level envelope and still owes the LLM-specific record series. The pragmatic pattern is to add the LLM gateway to the AI routing leg behind the existing API gateway.
- How does the LLM gateway handle the request to a streaming AI endpoint?
The streaming endpoint emits chunks over a single HTTP connection. The LLM gateway reads the request body at the start of the stream for the identity, classification, and policy decisions. The gateway reads the response chunks as the model emits them, applies the response-side classification and redaction per chunk, and commits the audit record at the end of the stream with the full request-response pair. The streaming pattern adds typically under 50 ms of overhead on the first chunk and below 5 ms per subsequent chunk.
- Does the LLM gateway require schema validation on the prompt body?
The OpenAI, Anthropic, and Bedrock provider APIs all publish OpenAPI specs. The LLM gateway can validate the envelope (the messages array shape, the model field, the parameters) against the provider's spec. The semantic content of the prompt is not schema-validatable in the OpenAPI sense. The gateway's classification and policy decisions cover the semantic side. The two together produce the validation surface the deployer expects.
- How does the gateway handle multiple model providers with different request shapes?
The LLM gateway carries an adapter per provider that normalizes the request and response shapes into the gateway's internal representation. The internal representation is what the policy engine evaluates against. The audit record series stores the provider identifier and the original request and response shape (or content-addressed reference to them) so the regulator can read the record without learning the gateway's internal representation. A deployer running OpenAI, Anthropic, and a self-hosted Llama deployment runs three adapters on the gateway and a single policy engine.