Databricks AI Gateway Security: The Governance It Adds Inside the Workspace and the Layer Beyond It

A Databricks AI Gateway call is an HTTPS POST to a model serving endpoint under https://{workspace}.cloud.databricks.com/serving-endpoints/{endpoint}/invocations, authenticated with a Databricks token and governed by the Mosaic AI Gateway configuration attached to that endpoint. Everything the gateway does operates on the endpoints it fronts inside the workspace: rate limiting, usage tracking, payload logging, and content guardrails. What it does not evaluate is whether the authenticated caller behind the token is permitted to send this particular prompt to this model, and it does not see a request that goes straight from an application to a non-Databricks LLM. I want to walk the controls Databricks ships, mark where they stop, and show the authorization layer that covers the part they leave to you.

The controls Databricks AI Gateway ships

Databricks gives a data team a real governance layer over its serving endpoints, and configuring it is the baseline.

Rate limiting and usage tracking. You set per-endpoint and per-user request limits, and the gateway meters token usage so a runaway job or a noisy consumer does not exhaust an upstream quota.

Payload logging via inference tables. The gateway can capture request and response payloads into a Delta inference table, giving a data team a queryable record of what went through an endpoint.

Guardrails. Configurable guardrails screen prompts and responses for content categories and can detect and mask PII patterns before a payload reaches or leaves the model.

Unity Catalog permissions. Access to serving endpoints is governed through Unity Catalog, so a principal needs an explicit grant to invoke an endpoint.

Where the Databricks controls stop

Each control has a boundary that matters for a real deployment. Unity Catalog grants coarse access to invoke an endpoint, and it does not reason about the content of a given request. Rate limits cap volume and never inspect meaning, so a caller under the limit can still send data that should never reach an external model. PII guardrails match patterns and miss the context that makes a clean-looking prompt sensitive, such as an unreleased financial figure that trips no PII regex. Inference tables record payloads inside the same platform that served the request, which is convenient telemetry and weaker independence than an audit demands. The whole layer covers endpoints Databricks fronts, so an application that calls OpenAI or Anthropic directly, outside the workspace, flows through none of it.

The gap on AI request traffic

The gap sits after authentication. The Databricks token confirms who is calling and Unity Catalog confirms they may invoke the endpoint. Neither answers whether this caller, in this role, is permitted to send this prompt, carrying this data classification, to this model right now. A data analyst with a grant on a serving endpoint can paste a table of customer records into an invocation, and every Databricks control passes it: the token is valid, the grant exists, the rate limit is fine, and the inference table dutifully logs the payload. This is the post-authentication gap, an authorization problem that lives at the request layer, above what the gateway inspects, and it widens the moment traffic leaves the workspace.

What identity-aware authorization on the request looks like

Closing the gap means evaluating the request itself, on the path to the model, against the caller's identity and the data in the prompt. In practice that is a policy decision point in front of the endpoint. A raw call looks like this:

An identity-aware enforcement layer intercepts that request, reads the caller identity, classifies the prompt content, evaluates policy, and either forwards the call or returns a decision like this before the model is reached:

The evaluation is deterministic and fail-closed, and it commits that record independently of the calling application before any response returns.

Compliance framing

For regulated workloads on Databricks, an inference table is not the audit record a regulator wants. EU AI Act Article 12 requires automatic, traceable event logging for high-risk systems, including inputs and the natural persons involved, applying from August 2, 2026 per the European Commission. A payload table written by the serving platform, for its own view of the call, sits in a weaker position than an identity-bound record produced independently of the application under review. The self-attestation problem I keep running into is simple to state: the system under audit should not be the system writing the audit.

What identity-aware enforcement does not remove

Adding an authorization layer does not retire the gateway's governance value. Rate limiting, Unity Catalog grants, and inference tables remain useful inside the workspace. The authorization decision runs above them, on the request, and it extends to the traffic Databricks never sees. A team running models both inside and outside Databricks needs one enforcement layer that covers both paths, not two partial ones.

DeepInspect

This is exactly what DeepInspect does. DeepInspect is a stateless proxy that sits inline in front of your AI traffic, and it is model-agnostic, so the same layer covers a Databricks serving endpoint, an OpenAI or Anthropic call, a Bedrock or Vertex request, and a self-hosted endpoint in one deployment. For every request it evaluates the caller's identity and role, the data classification in the prompt, and organizational policy, then returns a pass or block decision before the traffic reaches any model. Enforcement adds under 50 milliseconds in internal testing, invisible against the model's own response time, and the default on ambiguity is deny.

For every decision it commits an identity-bound audit record before the response returns, written independently of the calling application. That record carries the identity, policy, data classification, and outcome that an inference table never held. DeepInspect does not replace Unity Catalog grants, rate limits, or Databricks guardrails. It adds the authorization-and-audit layer that sits above them, across every model endpoint your workloads reach. Book a demo today.