How is this different from an API gateway?

An API gateway manages inbound traffic to services you host, handling routing, rate limits, and keys for your own APIs. An identity-aware proxy for LLMs governs the other direction, the outbound calls your apps and agents make to models you do not operate, and its central question is whether a named identity may send a given prompt to a given endpoint. Some gateways can be extended toward this, but the identity binding and prompt-level policy are what define the LLM pattern, and they are not what a conventional gateway was built to do.

Does the proxy need to store session state?

The common design is stateless, which means the proxy holds no per-session data between requests. Each call carries the identity context it needs, the proxy resolves policy and writes a record, and nothing has to be synchronized across proxy instances. Statelessness is what lets the enforcement point scale horizontally with traffic and keeps it from becoming a fragile piece of shared state in the request path.

Where does the identity context come from?

From your existing identity system, propagated into the request so the proxy can read who the authenticated user or agent is. The application continues to authenticate people and agents the way it already does; what changes is that the identity travels with the model call instead of being dropped at the shared key. For autonomous agents, the same mechanism carries an agent identity and scope rather than a human one.

Can it block a prompt, or only log it?

Both. Because the proxy terminates the request and reads the body before the model sees it, policy can permit the call, redact specific fields, or deny it outright. Logging is the record of whatever was decided. A design that could only log after the fact would leave the sensitive prompt already delivered, so acting inline, before the model responds, is central to the pattern rather than an add-on.

What happens to latency-sensitive applications?

The enforcement decision adds a fixed cost measured under 50 ms in production tests, set against model inference that runs from 500 ms to 5 seconds. For a user-facing AI feature, the perceived latency is dominated by generation, so the identity and content check is not the bottleneck. For high-throughput batch traffic, a stateless proxy scales out horizontally, so the per-request cost stays flat as volume grows.

Identity-Aware Proxy for LLMs: Putting a Named Caller on Every Model Request

A classic identity-aware proxy sits in front of a web application, checks who you are on every request, and decides whether you reach the resource behind it. The pattern moved access control off the network perimeter and onto the individual request. An identity-aware proxy for LLMs applies the same idea to a different kind of resource: the model API your applications and agents call, where today the request usually arrives under a shared key that names no one.

I want to walk through how the pattern works when the thing behind the proxy is a model rather than a web app, and why the shared-key habit is what makes it necessary.

The mechanism at the request layer

The proxy sits in the path between the caller and the model. A request comes in carrying, alongside the prompt, the identity context of the user or agent on whose behalf it is made. The proxy reads that context, resolves the caller's role and authorization, and evaluates the request against policy scoped to that identity and that route. Only then does the request continue to the model, unchanged, redacted, or blocked.

Because the proxy terminates the call rather than waving it through by destination, it sees the full request body. That is what lets policy act on content, not just on the host being reached. The decision resolves inline, before the model responds, and it writes a record of what it decided.

Why a shared key forces the issue

Most enterprise model traffic authenticates with a static API key issued to an application. On the wire, every service and every person behind that key looks the same. The model provider sees one account. Your own logs see one caller. The result is that authorization is binary and collective: anyone who can reach the key can send anything to the model, and no record distinguishes them.

An identity-aware proxy breaks that collapse. It reinserts the named caller into a flow that otherwise erases identity at the first hop. The application can still hold the provider key, but the proxy is what decides, per request, whether this identity may send this content to this endpoint. My view is that a shared model key without an enforcement point in front of it is not an integration, it is an unscoped grant to everyone who can find the key.

How it differs from a web-app identity-aware proxy

The web-app version answers "may this person open this page." The LLM version answers a richer question, because the sensitive payload is inside the request the user is sending, not only in the resource they are trying to reach.

That changes the work. The proxy has to classify the prompt for PII, secrets, and regulated data, because the exposure is in the outbound body. It has to apply per-route and per-role rules that can permit a call for one data class and deny it for another. And it has to produce a per-decision record suited to audit rather than a simple access-granted event. The identity check is the front of the pattern; content policy and the audit record are what the LLM setting adds.

What the pattern produces

Three outcomes fall out of putting an identity-aware proxy in front of models.

Attribution becomes possible. Every model call carries a named caller, so the question of who sent a given prompt has an answer that does not depend on guessing from a shared key.

Policy becomes per-identity. The same endpoint can be open to one team for one purpose and closed to another, and the decision is made at call time rather than baked into which service happens to hold the key.

Evidence becomes a byproduct. Each decision writes identity, role, policy version, classification, outcome, and timestamp, so the audit trail exists without a separate logging project.

Where it sits and what it costs

The proxy belongs in the request path such that every sanctioned route to a model passes through it and direct egress to model domains is closed at the network layer. A stateless design is the common choice, because it holds no session state to synchronize and scales horizontally with request volume.

The latency question comes up immediately, and the budget is comfortable. Enforcement overhead measures under 50 ms in production tests, while the model's own inference runs from 500 ms to 5 seconds. The identity resolution and content check resolve inside the time the user already waits for a response.

DeepInspect

DeepInspect is an identity-aware proxy for LLM traffic. It runs as a stateless proxy between your applications, agents, and staff and any model, reads the identity context the caller supplies, and evaluates each request against per-route and per-role policy while detecting and redacting PII and regulated data in the prompt.

Every decision produces a tamper-evident, per-decision record holding identity, role, policy version, data classification, outcome, and timestamp, committed before the model's response returns. The shared-key blind spot closes because the proxy, not the key, decides what each named caller may send.

If your models are reachable today through a shared key with no per-caller policy in front of them, book a platform demo and we will show you the pattern running against your own routes.