What does an open source AI gateway do?

An open source AI gateway is an abstraction over model providers. It gives application code one API in front of many models, handles routing and fallback across endpoints, applies rate limiting and caching to control spend and load, centralizes provider key management, and exposes observability such as token usage and latency. Projects in this category, including LiteLLM and the Envoy AI Gateway, focus on the operational problem of talking to many providers consistently. They are effective at that layer, which covers most of the integration work a platform team faces when standing up model access.

Is an open source AI gateway enough for compliance?

For operational needs, often yes; for regulatory record obligations, usually not without additional work. Open source gateways focus on routing, keys, and metrics, and they typically leave identity-bound authorization, prompt-level classification, and tamper-evident per-decision audit records as extension points. Compliance regimes ask for a record of what the AI did with specific data, who authorized it, and under what policy, which usage metrics do not provide. A regulated deployment can keep the open source gateway for provider integration and add a control-and-audit layer in front of it to meet the record and authorization requirements.

Are open source AI gateways secure?

They can be operated securely, and like any component that authenticates callers and stores provider keys, they carry their own attack surface. The June 2026 LiteLLM CVE wave, which included an authentication bypass and a remote-code-execution flaw added to CISA's Known Exploited Vulnerabilities catalog, is a reminder that a gateway's auth and key-storage layer is a high-value target. Evaluating an open source gateway for security means looking at how it authenticates identities, how it stores secrets, and how quickly it patches, in addition to the traffic controls it applies. Key custody and identity handling are security properties, not only operational features.

Can DeepInspect work with an open source AI gateway?

Yes. DeepInspect operates at the control layer and composes with an operational gateway rather than replacing it. A deployment can keep its open source gateway for provider integration, routing, and caching, and place DeepInspect at the AI request boundary in front of it to enforce identity-bound authorization, classify and redact prompt data, fail closed on unevaluable requests, and produce a signed per-decision audit record. The open source gateway handles the operational job of reaching many providers, and DeepInspect handles the control and evidence job that regulated deployments require. The two run together as separate layers.

Open Source AI Gateway: What It Covers and Where the Gap Is

Open source AI gateways solved a real operational problem. Talking to OpenAI, Anthropic, Bedrock, and a self-hosted model used to mean four SDKs, four auth schemes, and four retry strategies. Projects like LiteLLM and the Envoy AI Gateway collapsed that into one interface: a unified API, provider routing, rate limiting, caching, key management, and request metrics. For a platform team standing up model access, that is most of the operational work done.

I want to be clear about what the open source category covers, because it is genuinely strong at the operational layer, and equally clear about the control layer it mostly leaves to you.

What the operational layer gives you

An open source AI gateway is an abstraction over model providers. The value it delivers is consistent across the well-known projects:

One API surface in front of many providers, so application code stops caring which model it calls.
Routing and fallback across endpoints, including cost-based and latency-based routing.
Rate limiting and caching to control spend and load.
Key management so provider credentials sit in one place instead of scattered across services.
Observability: request counts, token usage, latency, error rates.

That is a solid foundation, and for many deployments it is enough. The reason it is worth naming precisely is that the list describes operations, not control. Nothing in it decides whether a particular identity was permitted to send a particular prompt to a particular model, and nothing in it produces the kind of record a regulator asks for.

Where the control layer begins

The control layer is a different set of questions, and open source gateways address it partially or leave it as an extension point:

Identity-bound authorization. Routing decides where a request goes. Authorization decides whether this identity may send it at all. Most open source gateways authenticate a caller with a virtual key that maps to a team or budget, which is an operational identity, not the natural-person identity an audit trail needs.
Prompt-level classification and redaction. Detecting PHI, NPI, or secrets inside prompt content and acting on it is usually a plugin you write or wire in, not a built-in decision.
Tamper-evident per-decision audit. Request metrics are not an audit record. An audit record ties identity, data classification, policy version, and outcome to each decision and resists modification by the system that produced it. I walked through why that independence matters in stateless vs stateful AI proxies.
Fail-closed policy. An operational gateway optimizes for uptime and tends to fail open. A control fails closed. That default is a design decision the deployment has to make deliberately.

None of this is a criticism of the projects. They were built to solve the provider-integration problem, and they solve it. The control layer is a separate concern that a regulated deployment has to add.

The gateway's own attack surface

There is a second reason to look closely at the control layer, and the June 2026 LiteLLM CVE wave illustrates it. Over one weekend, multiple LiteLLM vulnerabilities were disclosed, led by an authentication-bypass in the key-auth function, on top of an earlier remote-code-execution flaw that CISA added to its Known Exploited Vulnerabilities catalog. The lesson is architectural rather than a knock on one project: a gateway that stores provider keys and authenticates callers is itself a high-value target. Its auth and key-storage layer is an attack surface, so the way it handles identity and secrets is a security property, not just an operational convenience.

This is where the OWASP GenAI guidance on authenticating model access and the June LiteLLM CVE reporting summarized by security press point the same direction: evaluate the gateway's own authorization and key custody as carefully as the traffic it carries.

How to evaluate an open source gateway for regulated use

Three questions separate an operational gateway from a control-grade one. Does it bind requests to a verified natural-person identity, or only to a team-level virtual key. Does it produce a per-decision audit record that is independent of the application and resistant to tampering, or only usage metrics. Does it fail closed when it cannot evaluate a request, or fail open to preserve availability. Where the answers are "extension point" or "not built in," that is the work a regulated deployment has to own, and I mapped the category lines in AI firewall vs AI gateway vs AI proxy.

DeepInspect

DeepInspect operates at the control layer, and it composes with the operational one. DeepInspect is a stateless proxy that sits at the AI request boundary between your users or agents and any LLM endpoint. It evaluates each request against the identity the application supplies and per-route, per-role policy, classifies the data in the prompt, and decides permit, redact, or deny inline, failing closed when it cannot evaluate. It holds no long-lived provider keys as a persistent store for an attacker to reach.

Every decision produces a signed, per-decision audit record: identity, data classification, policy version, outcome, and timestamp, committed before the response returns to the application. A deployment can keep its open source gateway for provider integration and routing and place the identity-bound enforcement and audit layer in front of it, which is the AI control plane role. The operational layer and the control layer do different jobs, and they run together.

If you are running an open source AI gateway and need identity-bound authorization and tamper-evident audit records for a regulated deployment, book a demo today.