On-Prem LLM Security: A Deployment Acceptance Matrix

An on-premises LLM deployment creates several security boundaries before the first prompt reaches the model. The GPU host has an operating system and drivers. The cluster has identities and secrets. It also has images and network paths. The inference server exposes endpoints with its own assumptions. Model weights and caches need custody controls. Only then does the application route an authenticated request. I would block launch until one acceptance sheet names the owner and test for every exposed interface. The evidence should sit beside them.

TL;DR

• Treat the host and cluster as separate control planes. Do the same for the inference server and model assets. Keep the request route separate too.
• Give each exposed interface an owner and required control. Add an acceptance test and retained evidence before launch.
• Put vLLM distributed traffic on trusted internal networks and place a controlled reverse proxy in front of allowed endpoints.
• Apply DeepInspect only to authenticated HTTP model calls deliberately routed through it; host and cluster controls remain separate.

The acceptance unit is an exposed interface

An on-prem LLM security review should begin with interfaces rather than product names. Draw boxes for the user-facing application and request gateway. Add one for the inference endpoint and one for each distributed worker or storage service. Add the Kubernetes API and container registry. Include the model repository and cache. Then add the monitoring collector and administrative access path.

The vLLM security guidance warns that distributed communication interfaces assume a trusted internal network. It also explains that API-key protection does not cover every endpoint and recommends a reverse proxy for endpoint allowlisting and authentication. It also recommends rate limiting and logging. Those details turn a generic deployment diagram into a list of testable surfaces.

Put the diagram on a plain monitor and follow one request with a finger. Every point where the request changes protocol or identity needs a row in the matrix. A change of owner needs one too. A line without a named owner is an unresolved launch condition.

Use one matrix across five control planes

The matrix needs seven fields:

• Component: Specify the host or cluster service. Name the inference endpoint or model store where applicable. Include any routed application surface.
• Exposed interface: Record the network listener and API. Add any administrative path or file store. Include each registry operation.
• Owner: the team that can change and operate the control.
• Required control: the specific condition needed for launch.
• Acceptance test: the action that proves the condition under documented circumstances.
• Evidence: Retain the configuration or event. Keep any denial or recovery artifact produced by the test.
• DeepInspect applicability: yes only when the interaction is authenticated HTTP model traffic deliberately routed through the gateway.

This structure prevents one gateway log from being offered as proof that a GPU host was patched or a Kubernetes secret was protected. I would reject any acceptance packet that uses one green dashboard to stand in for five independent control planes.

Host and cluster rows come first

The GPU host row should name the operating-system baseline and the driver and runtime versions. It should also name the administrative identities and patch owner. Document the disk protection and monitoring path. Include the recovery procedure. Test administrative access with the approved identity, then record the blocked path for an unauthorized identity and retain the baseline used during the test.

The Kubernetes security checklist supports separate rows for role-based access control and workload identity. It also covers network policy and audit logging. Add rows for secrets and image controls, along with public exposure. It recommends default-deny network policies. It also warns against exposing the Kubernetes API or kubelet API publicly. The same warning applies to etcd. The checklist also calls for restricted access to sensitive components and review of permissions.

Translate those recommendations into local tests by attempting an unauthorized namespace read and confirming that a workload cannot reach an unapproved destination. Verify that the API and etcd have no public listener. Then pull an unapproved image and retain the rejection. The resulting audit event should identify the acting principal and operation.

Cluster networking and host execution remain infrastructure controls. The same is true of images and secrets. Local administrative actions also remain infrastructure controls even when the deployment serves an LLM.

Inference-server controls need endpoint-level tests

Treat the inference server as a service with a precise endpoint contract that lists every listener and route. Identify which endpoints require authentication and which remain internal. Record which endpoints the reverse proxy allows. Add request-size and concurrency limits. Define the timeouts and logging behavior, along with failure handling.

For vLLM, keep distributed worker traffic inside the trusted network described by its documentation. Put the approved public or application-facing endpoints behind the reverse proxy pattern. Test a permitted route with the expected identity. Then request an unapproved administrative or diagnostic route and retain the denial.

A single API key is weak acceptance evidence when nearby endpoints do not share that protection. The matrix should show each exposed route and the control that receives it. Record the proxy configuration version beside the resulting access event so another operator can reproduce the decision.

The self-hosted Llama gateway patterns guide covers request controls for a Llama endpoint. The broader on-prem matrix keeps that route work beside the infrastructure and custody controls needed for the complete deployment.

Model assets need custody and integrity evidence

Model weights and adapters enter the deployment through supply paths. So do tokenizer files and prompt templates. Container images use supply paths too. Name the approved source and version for each asset. Record who can promote it and where integrity information is checked. Keep staging and production references separate.

Supply-chain validation starts with the exact artifact selected for release. Confirm its approved origin and digest. Attempt to substitute an unapproved version through the normal promotion path, then retain the rejection. Prove that the running workload references the approved artifact rather than a mutable nickname.

NIST Special Publication 800-53 Revision 5 provides control families for access control and identification and authentication. It also covers audit and accountability, along with configuration management. Other families cover incident response and communications protection. They also cover system integrity and supply-chain risk management. Use those families to organize evidence while keeping the deployment tests specific to the local architecture. The publication is a control catalog, not an on-prem LLM installation recipe.

Route policy starts after identity is known

The application-facing row should identify the caller and destination model route. It should also state the request class and policy expected for that combination. Test one allowed request and one prohibited request with known identities. Retain the route and policy version. Keep the decision and timestamp. Record the response outcome.

DeepInspect can serve this row when the application deliberately sends authenticated user or agent traffic over HTTP through the gateway before the LLM call. It can evaluate the caller and request against the active policy and record that decision outside the application's own write path.

A local shell command or Kubernetes admission decision remains outside that control. So does a model-file read or distributed-worker message. A STDIO tool call or inference request that bypasses the gateway is also outside it. Each event belongs to the matrix row for the system that receives it.

Failure and recovery need their own evidence

Acceptance should include a failed dependency and an unavailable policy point. Define whether the application fails closed or stops the request path. Document any approved degraded mode separately. Trigger the documented condition in a controlled environment and retain the observed result.

Recovery evidence should identify the restored version and configuration. Replay the allowed and prohibited route tests after recovery. For cluster or host recovery, verify the relevant identity and network controls again. Verify the image and audit controls too, rather than assuming a restored service preserved them.

The AI gateway deployment patterns guide covers request-path placement and availability choices. The on-prem acceptance matrix links that route decision to infrastructure recovery without claiming that either one proves the other.

DeepInspect

DeepInspect fits one row in the on-prem LLM security matrix: the authenticated HTTP request path from a known user or agent to an LLM. The acceptance test can preserve the caller and route. It can retain the active policy and decision. It should also keep the timestamp and resulting model transaction outside the calling application's own log stream.

The infrastructure closure packet should contain the denied access and approved artifact digest. It should also contain the restored configuration and rerun evidence from the systems that produced them.

Book a technical deep dive at deepinspect.ai.