Databricks AI Gateway: Guardrails, Rate Limits, Inference Tables

A Databricks AI Gateway request reaches a Model Serving endpoint over HTTPS and is governed by the gateway configuration attached to that endpoint. Databricks documents AI Gateway controls for rate limits, usage tracking, inference tables, and AI guardrails. This article shows what each one records or blocks, then traces the additional authorization decision required when a caller sends a particular data class to a particular model.

TL;DR

Databricks AI Gateway governs Model Serving endpoints with limits, guardrails, usage tracking, and inference tables. Unity Catalog access authorizes endpoint invocation; a request-level policy layer evaluates the caller, prompt data, target model, and policy before the model receives routed traffic.

The controls Databricks AI Gateway ships

Databricks gives a data team a real governance layer over its serving endpoints, and configuring it is the baseline.

Rate limiting and usage tracking. You set per-endpoint and per-user request limits, and the gateway meters token usage so a runaway job or a noisy consumer does not exhaust an upstream quota.

Payload logging via inference tables. The gateway can capture request and response payloads into a Delta inference table, giving a data team a queryable record of what went through an endpoint.

Guardrails. Configurable guardrails screen prompts and responses for content categories and can detect and mask PII patterns before a payload reaches or leaves the model.

Unity Catalog permissions. Access to serving endpoints is governed through Unity Catalog, so a principal needs an explicit grant to invoke an endpoint.

Where the Databricks controls stop

Each control has a boundary that matters for a real deployment. Unity Catalog grants coarse access to invoke an endpoint, and it does not reason about the content of a given request. Rate limits cap volume and never inspect meaning, so a caller under the limit can still send data that should never reach an external model. PII guardrails match patterns and miss the context that makes a clean-looking prompt sensitive, such as an unreleased financial figure that trips no PII regex. Inference tables record payloads inside the same platform that served the request, which is convenient telemetry and weaker independence than an audit demands. The whole layer covers endpoints Databricks fronts, so an application that calls OpenAI or Anthropic directly, outside the workspace, flows through none of it.

The gap on AI request traffic

The gap sits after authentication. The Databricks token confirms who is calling and Unity Catalog confirms they may invoke the endpoint. Neither answers whether this caller, in this role, is permitted to send this prompt, carrying this data classification, to this model right now. A data analyst with a grant on a serving endpoint can paste a table of customer records into an invocation, and every Databricks control passes it: the token is valid, the grant exists, the rate limit is fine, and the inference table dutifully logs the payload. This is the post-authentication gap, an authorization problem that lives at the request layer, above what the gateway inspects, and it widens the moment traffic leaves the workspace.

What identity-aware authorization on the request looks like

Closing the gap means evaluating the request itself, on the path to the model, against the caller's identity and the data in the prompt. A policy decision point in front of the endpoint needs the following inputs before it forwards a call:

• Authenticated caller identity and role.
• Target serving endpoint and model.
• Prompt data classification assigned by policy.
• The applicable policy version and its pass or block result.
• A timestamp and correlation identifier for the resulting audit record.

When policy blocks the request, the caller should receive a decision that identifies the blocked policy and endpoint without exposing the original sensitive prompt.

The evaluation is deterministic and fail-closed, and it commits that record independently of the calling application before any response returns. NIST's Generative AI Profile provides useful risk-management context for making that decision reviewable.

Compliance framing

For regulated workloads on Databricks, an inference table is not the audit record a regulator wants. EU AI Act Article 12 requires automatic, traceable event logging for high-risk systems, including inputs and the natural persons involved, applying from August 2, 2026 per the European Commission. A payload table written by the serving platform, for its own view of the call, sits in a weaker position than an identity-bound record produced independently of the application under review. The self-attestation problem I keep running into is simple to state: the system under audit should not be the system writing the audit.

What identity-aware enforcement does not remove

Adding an authorization layer does not retire the gateway's governance value. Rate limiting, Unity Catalog grants, and inference tables remain useful inside the workspace. The authorization decision runs above them, on the request, and it extends to the traffic Databricks never sees. A team running models both inside and outside Databricks needs one enforcement layer that covers both paths, not two partial ones.

DeepInspect

This is exactly what DeepInspect does. DeepInspect is a stateless proxy that sits inline in front of routed HTTP AI traffic, so the same layer can cover a Databricks serving endpoint and other LLM endpoints that the deployment sends through the proxy. For every request it evaluates the caller's identity and role, the data classification in the prompt, and organizational policy, then returns a pass or block decision before the traffic reaches the model. The default on ambiguity is deny. See DeepInspect vs Databricks AI Gateway for the architectural comparison and Databricks AI Gateway alternatives for deployment contexts that need a different control surface.

For every decision it commits an identity-bound audit record before the response returns, written independently of the calling application. That record carries the identity, policy, data classification, and outcome that an inference table never held. DeepInspect does not replace Unity Catalog grants, rate limits, or Databricks guardrails. It adds the authorization-and-audit layer that sits above them, across every model endpoint your workloads reach. Book a demo today.