Does Azure API Management inspect the prompt content?

Not natively for security classification. API Management terminates TLS and can read the JSON request body, and its GenAI policies count tokens and emit cost metrics keyed to the calling subscription. It has no built-in understanding of what the messages content contains, so it cannot tell whether a prompt carries PII, source code, or regulated data. You can call out to Azure AI Content Safety or a custom classifier from a policy expression, which means you are building payload inspection on top of a gateway designed to govern the request envelope.

Should I use the resource key or managed identity to call Azure OpenAI?

Managed identity with Entra ID and the Cognitive Services OpenAI User role, per Microsoft's guidance. The resource key grants full access to every deployment on the resource and cannot be scoped or attributed per caller. In the gateway pattern, consumers authenticate to API Management with their own credential, and API Management authenticates outbound to Azure OpenAI with a single managed identity. That gives you per-caller revocation and attribution that a shared key structurally cannot provide.

How does Azure OpenAI content filtering relate to a gateway?

Azure OpenAI content filtering runs inside the model deployment through Azure AI Content Safety, scoped to provider-defined categories like hate, violence, and self-harm. It is a model-layer control, and it fires on the provider side after your request has already reached Azure. A gateway sits earlier, on the network path before the request leaves your boundary, which is where identity-bound policy and organizational data classification belong. The two layers address different questions and are worth running together.

Can one gateway front Azure OpenAI and other providers at once?

A network-layer gateway like API Management can proxy multiple backends, but its GenAI policies are written for the Azure OpenAI token and metric shape. A model-agnostic policy layer evaluates identity, data classification, and organizational policy the same way regardless of whether the backend is Azure OpenAI, Bedrock, or a self-hosted model. That matters in a mixed estate, because writing and maintaining separate policy logic per provider is where drift and gaps appear.

Azure OpenAI Gateway Patterns: Where the Governed Proxy Sits and What It Reads

An Azure OpenAI deployment lives at a resource-scoped endpoint, https://{resource}.openai.azure.com/openai/deployments/{deployment}/chat/completions, and every call carries either an api-key header or an Entra ID bearer token. The moment a second team wants to call the same model, someone stands up a gateway in front of it. On Azure that gateway is usually Azure API Management, and it does real work: it caps token spend, attributes cost per caller, and spreads load across several regional deployments. I want to walk the three patterns teams actually deploy, show the policy config that carries the weight, and be precise about where the request envelope ends and the prompt payload begins.

Why the resource key is the wrong sharing primitive

The Azure OpenAI resource key grants full access to every deployment on that resource. Hand it to three teams and you lose per-caller attribution, per-caller rate limits, and the ability to revoke one consumer without rotating the key for all of them. Microsoft's own guidance steers you to managed identity and Entra ID with the Cognitive Services OpenAI User role rather than shared keys. The gateway pattern formalizes this: consumers authenticate to API Management with their own subscription key or JWT, and API Management authenticates to Azure OpenAI with a single managed identity the consumers never see.

Pattern one: token governance and cost attribution

The first job is spend control. Azure OpenAI bills on prompt and completion tokens, and a single unbounded consumer can exhaust a provisioned throughput deployment for everyone. API Management ships two GenAI policies for exactly this, the azure-openai-token-limit policy and the azure-openai-emit-token-metric policy:

The counter-key binds the limit to the calling subscription, so one team hitting its ceiling never starves another. The emitted metric flows to Azure Monitor, which gives you cost attribution per consumer without parsing provider invoices. This is genuine governance of the request envelope: how many tokens, from which subscription, at what rate.

Pattern two: load balancing across regional deployments

The second job is availability. A single Azure OpenAI deployment has a fixed quota, and provisioned throughput units are regional. Teams put several backends behind one API Management API and let the gateway spread traffic, falling back when a region returns a 429. The backend pool with circuit breaker is the current mechanism:

Priority one takes traffic until its circuit breaker trips on repeated 429s, then priority two absorbs the overflow. This keeps a consumer online across a regional capacity limit. The gateway is routing by response status here, and the response status is metadata about the envelope. It reads nothing inside the completion.

Pattern three: the content-policy gap

The third job is where the pattern hits its edge. Teams reach for API Management to also enforce policy on what flows through the prompt, and here the tooling thins out. Azure ships content filtering inside the model deployment through Azure AI Content Safety, and that filter runs at the inference layer, scoped to categories the provider defines. The API Management layer can inspect the JSON body with a generic validate-content or a custom policy, but it has no native understanding of PII, PHI, source code, or jurisdictional data classes inside a messages array. You can bolt a call out to Azure AI Content Safety or a custom classifier, and now you are building a payload governance layer by hand on top of a gateway designed for the envelope.

What governing the payload requires

Governing the prompt payload means reading the decrypted messages content on every request, classifying it against organizational data categories, binding the decision to the caller identity the application supplied, applying a pass, block, or modify outcome before the request reaches {resource}.openai.azure.com, and committing a signed record of that decision. Azure API Management terminates TLS, so it sits on the decrypted HTTP path where this is possible in principle. What it lacks out of the box is the classifier, the identity-bound policy model, and the tamper-evident audit primitive. The attack tempo makes the timing non-negotiable: Google Mandiant's M-Trends 2026 report put the median handoff from initial access to a secondary actor at 22 seconds, so a control that reviews prompts after the fact provides forensic value and no prevention.

DeepInspect

This is the gap DeepInspect closes on Azure. Your API Management layer keeps doing what it does well: token limits per subscription, cost metrics to Azure Monitor, and backend pools that ride out a regional 429. DeepInspect adds the payload governance that sits above the envelope.

DeepInspect is a stateless proxy on the AI request path in front of your Azure OpenAI endpoint. It terminates the AI request, reads the decrypted messages body rather than the encrypted tunnel, binds the call to the identity your application passes through from Entra ID, classifies prompt content across PII, PHI, MNPI, source code, and jurisdictional categories, and applies a deterministic pass, block, or modify decision before the request reaches the deployment. It commits a per-decision audit record for every call, inline and fail-closed. It is model-agnostic, so the same policy covers Azure OpenAI, a self-hosted model, and a second provider in the same estate. Book a demo today.