A Self-Hosted AI Gateway Moves Control Inward Without Removing Provider Risk

A self-hosted gateway runs the control process in infrastructure you operate. The model can still sit at a public provider endpoint, receiving the prompt after the gateway permits it. Hosting location changes control custody. It does not erase the downstream route.

TL;DR

• Self-hosting gives the enterprise custody of gateway configuration, credentials, policy code, logs, and upgrade timing.
• External model traffic still leaves the environment unless the selected endpoint is also private or on premises.
• Test identity propagation and log durability, then test bypass and fail-closed behavior before production approval.

What moves into your environment

LiteLLM documents a self-hosted proxy deployment that centralizes model access and configuration. Kong's AI Gateway documentation describes model routing, provider integrations, observability, and policy features within its gateway platform.

The exact feature sets differ, but the operational transfer is similar. Your team owns the runtime, network path, secret store integration, database, scaling, patching, and telemetry destination. A managed gateway vendor owns more of that stack. A self-hosted label says where the gateway runs, rather than where every dependency runs.

That transfer changes the evidence required for approval. The platform team must show where configuration lives and how a change reaches production. Security reviewers also need the path used by provider credentials and the location of each decision record. Those details establish which controls actually moved inward. They also expose dependencies that the self-hosted label leaves outside the environment.

Custody creates work. Capacity limits and certificate renewal now depend on the enterprise operating process, while the provider still controls the hosted model endpoint. Review both boundaries separately. Combining them under one private deployment claim makes it harder to assign failures to the system that caused them.

Egress is the decisive map

Draw the route on one page. Mark the user or agent, application, gateway, DNS resolver, outbound firewall, provider endpoint, and log sink. Then mark every TLS termination point and credential boundary. A rack diagram with a bright green "private" box around the gateway can hide the arrow leaving for a public model API.

Our AI gateway architecture guide explains the routed boundary. If a self-hosted model is the destination, the request can remain inside controlled infrastructure. If the destination is a hosted provider such as OpenAI or Anthropic, the approved request still crosses to that provider under its contract and retention terms.

The drawing should follow one request from caller to response. Include the rejected path as well, because a denied request can still generate a decision record even though no prompt reaches the model. Compare that route with the application's network permissions. If the application can reach the provider without passing through the gateway, the diagram has documented a bypass rather than removed it.

Egress review also needs a failure path. Show what happens when name resolution fails, the provider endpoint times out, the outbound firewall blocks the request, or the gateway cannot write its record. Each outcome should have an owner and an observable signal. A generic box labelled "network" cannot provide either.

Four production tests

First, remove caller identity and confirm the gateway denies the request. Second, break the policy service and verify fail-closed behavior. Third, call the model endpoint directly from the application subnet and confirm network controls block the bypass. Fourth, crash the application after the provider responds and check that the independent decision record still exists.

Run each test against the same route the production application will use. A result from a separate test subnet says little about the outbound rule attached to the real workload. Record the request time and the gateway decision so the security team can match the attempted call to the durable evidence.

Repeat the bypass test after a deployment change. Network rules can drift independently of gateway policy, and a successful policy test does not prove that direct provider access is blocked. Treat the two checks as separate controls. One proves enforcement on routed traffic; the other proves that traffic cannot choose a different route.

I would delay launch if direct provider egress is open. A gateway that applications can bypass is an optional library wearing an infrastructure badge.

Operations become part of security

Self-hosting adds upgrade and capacity decisions. A queue spike, expired certificate, full disk, or database outage can alter enforcement availability. Define who patches the gateway, who reviews policy changes, who approves upgrades, and who can read prompt-bearing logs. Separate those roles where the platform permits it.

Write the failure behavior before an outage. The team needs to know which requests stop, where operators see the condition, how the missing capacity is restored, and who approves recovery. That operating decision affects enforcement just as directly as a policy rule. It deserves the same review trail.

Upgrade timing needs its own evidence. Record the deployed version and the approval for each change, then verify the policy path after installation. The point is accountability. Self-hosting gives the enterprise control over timing, so an unpatched instance or an untested upgrade is also an enterprise decision.

The identity-aware gateway guide covers caller context. Service credentials authenticate the gateway to the provider; they do not identify the employee or agent behind one request.

DeepInspect

DeepInspect can run as the enforcement layer on authenticated HTTP traffic to approved LLM endpoints. It evaluates identity and content policy before forwarding and writes a per-decision record independent of application code.

Book a technical deep dive at deepinspect.ai.