AI Inference
AI inference is the process of running a trained model against new input to produce an output: the moment a large language model turns a prompt into a completion, or a classifier turns an image into a label. Training builds the model's weights. Inference uses those fixed weights to serve predictions in production. For LLM applications, inference is where user data meets the model, usually as an HTTP request to a hosted inference endpoint. That request-and-response exchange is the point where prompts carry sensitive data out and model outputs carry risk back in.
How inference runs in production
A production inference call is almost always an HTTP request to a model endpoint: the application sends a prompt (plus system instructions and any retrieved context) to a URL, and the endpoint streams back generated tokens. Whether the endpoint is a commercial API or a self-hosted server, the shape is the same, a request payload in and a completion out. The volume and speed of these calls, thousands per minute in an active deployment, is why inspection at inference time has to add minimal latency.
Why the inference call is a control point
Everything that reaches the model passes through the inference request, and everything the model says passes back through the response. That makes the inference call the natural place to classify prompt content, enforce which identity may send which request to which model, and capture a record of the exchange. DeepInspect sits in front of the inference endpoint as a stateless proxy, evaluates identity-aware policy on each request, and logs a per-decision audit record, all on the HTTP path so the model provider integration stays unchanged.
Related reading
- Securing the Inference Lifecycle: The Five Stages Where the Enforcement Layer Has To Sit
The AI inference lifecycle is the sequence the application runs every time the model produces a response. Most security programs cover model training and the post-deployment monitoring stages but leave the inference path itself uninstrumented. This piece walks through the five stages of the inference lifecycle, the control points each stage exposes at the request boundary, the per-decision audit record the deployment has to commit, and the architectural pattern that closes the inference-time gaps a 2022-era AppSec program leaves open.
- The LLM inference gateway: what sits between authenticated callers and the model, and what belongs somewhere else
The LLM inference gateway is the identity-aware policy enforcement point between authenticated users or agents and any model endpoint. It is the layer where authorization, data classification, and audit-record production live. This piece defines the term, walks through the four fields the gateway resolves per request, contrasts it with the inference server, model router, and API gateway it is often confused with, and shows why the audit-write path must be isolated from the caller. Applies to any deployment running an OpenAI-compatible or provider-native LLM API in production.