Does self-hosting Llama remove the need for a gateway?

No. Self-hosting removes the third-party provider from the data path, which answers the external-egress question. It leaves the internal access-control question open. A self-hosted inference server still exposes an HTTP endpoint that accepts a prompt from any process that can reach it, with no built-in notion of which caller is authorized or what data class belongs in the prompt. A gateway supplies the identity binding, the prompt classification, and the audit record the raw inference port lacks.

Where is the boundary between the gateway and the inference host?

The gateway governs the HTTP request and response between callers and the model. The host that runs vLLM or TGI, the model weights on disk, and the container itself sit outside that boundary and are infrastructure-security concerns. A process that has already gained local execution on the inference host is also outside the boundary, because that traffic never crosses the network path the gateway inspects. Treat the gateway as the control on who may call the model and with what, and secure the host separately.

How does a gateway work with vLLM or Ollama?

Both expose an OpenAI-compatible HTTP API. You point client applications at the gateway's base URL instead of the inference port, give each client a per-identity token, and bind the vLLM or Ollama port to an address only the gateway can reach. The gateway verifies the token, applies policy, and forwards to the inference server using its own internal route. From the application's side the change is the base URL and the credential.

Is prompt classification still useful when the model is internal?

Yes. The data sensitivity that justifies self-hosting is the same reason to inspect prompts. An internal model still receives whatever a caller puts in the context window, and a caller authorized for one task can include regulated data it has no basis to send. Classifying the outbound prompt against data classes and enforcing policy on the class keeps an internal endpoint from becoming an unmonitored channel for the exact data the self-hosting decision was meant to protect.

Self-Hosted Llama Gateway Patterns: Governing an Inference Endpoint You Own

A self-hosted Llama deployment is an HTTP service. You run the weights behind vLLM, TGI, or Ollama, and each of those exposes an OpenAI-compatible endpoint at /v1/chat/completions. Any process that can reach that port sends a prompt and gets a completion back. The model is on your hardware, in your network, under your control. The request path into it is still an open HTTP door.

Teams move to self-hosted Llama to keep prompts off a third-party provider, and that decision does close the external-egress question. It leaves the internal ones open. Who is allowed to call this endpoint. What class of data is allowed in the prompt. What record exists after the call. Those are the same identity, authorization, and audit questions a hosted model raises, and running the weights yourself answers none of them.

What the gateway governs, and what it does not

A gateway in front of a self-hosted Llama endpoint sits on the HTTP path between callers and the inference server. It governs the request and the response. It does not govern the box.

Inside the boundary:

Any application, agent, or user process that calls the inference server over HTTP.
The prompt content in that call and the completion that returns.

Outside the boundary:

The GPU host, the model weights on disk, and the container the server runs in. Those are infrastructure-security concerns.
A process that has already shelled into the inference host. Local execution never crosses the gateway.
Model behavior itself. A self-hosted model produces the same probabilistic output any model does, and shaping that output is a separate job from governing who may call it.

The honest pattern is a gateway on the HTTP boundary plus real controls on the host and the network fabric around it. The gateway makes the endpoint governed. It does not make the server secure on its own.

Put every caller behind a verified identity

The default self-hosted setup ships one shared API key, or no key at all on a trusted network. Every internal service that calls the endpoint looks identical to the server, so a review of who sent a given prompt ends at an IP address or a single static token.

Route callers through the gateway and have each present an identity the gateway can bind to a request:

The gateway verifies the token, evaluates policy against that caller identity, and forwards to the vLLM or TGI endpoint on an address only the gateway can reach. Now a decision record names the reporting service, not a shared secret. This is the identity-aware property that a raw inference port lacks.

Classify the prompt against the data you are trying to keep in-house

The reason to self-host is usually the sensitivity of the data. That reason is also why prompt classification still matters after the model is internal. A caller can put regulated data into a prompt whether the model runs on your GPU or someone else's, and internal does not mean unrestricted. The gateway inspects the actual outbound prompt against regulated data classes and applies policy on the class, so a service authorized for general summarization gets blocked when its prompt carries PII it has no basis to send to the model, even an internal one.

Force every call through one path with egress control

A per-caller identity means little if a service can open a direct socket to the vLLM port and skip the gateway. Bind the inference server to an address the gateway owns, and restrict egress from the application tier so the only route to the model is through the enforcement point:

With the inference port reachable only from the gateway, a misconfigured client or an injected instruction that tries to hit the model directly fails at the network. The egress control keeps the gateway as the single place every prompt passes through, which is the precondition for both policy and a complete audit trail.

Record every decision on a path the caller cannot edit

Each call to the self-hosted model produces a decision record: the caller identity, the data class in the prompt, the pass-or-block outcome, the policy version, and a timestamp. The record commits before the completion returns, on a write path the calling application never has custody of. That write-path independence is what separates an audit record from an application log the same service could selectively write or drop on a crash. For a self-hosted deployment that exists precisely because the data is regulated, that record is the evidence that the internal model was used within policy.

DeepInspect

This is exactly what DeepInspect does. A self-hosted Llama endpoint answers HTTP, and DeepInspect sits inline between your callers and that endpoint, the same way it sits in front of any hosted LLM API.

For every request, DeepInspect binds the call to the identity the application supplies, classifies the prompt against regulated data types, applies per-caller and per-route policy, and makes a deterministic, fail-closed decision before the request reaches the inference server. Every decision commits a signed, identity-bound audit record before the completion returns, on a write path the caller never controls. DeepInspect is model-agnostic, so the same policy set covers your self-hosted Llama, a self-hosted Mistral, and a hosted provider in one deployment. It governs the HTTP traffic to the model and does not claim to secure the GPU host or the weights on disk, which stay an infrastructure job. If you have stood up an internal Llama endpoint and cannot say which caller sent which class of data to it, book a demo today.