Inference Endpoint
An inference endpoint is the network address, almost always an HTTP API, where an application sends input to a deployed model and receives the model's output. It is the production interface to a large language model: the application POSTs a prompt to the endpoint's URL and reads back generated tokens. Inference endpoints can be commercial APIs from a model provider or self-hosted servers exposing an open-source model. Every AI request in a deployment terminates at some inference endpoint, which makes the set of endpoints a workload can reach a core question for both security and cost.
How an inference endpoint is used
An application authenticates to the inference endpoint (usually with an API key or token), sends a request containing the prompt, system instructions, and parameters, and receives a completion, often streamed token by token. A single deployment may talk to many endpoints: different providers, different models, different regions. The endpoint URL and the credential together determine which model answers and under whose account the call is billed and logged.
Why the endpoint is where policy belongs
Because every prompt and every response passes through the inference endpoint, the request to that endpoint is the one place where all AI traffic can be seen and governed. Placing an enforcement layer in front of the endpoint means identity, payload, and destination are evaluated on the same HTTP path the model call already uses. DeepInspect operates as a stateless proxy at that position, applying per-identity policy and writing a per-decision audit record for each call, without requiring changes to the model provider or the application's request shape.
Related reading
- The LLM inference gateway: what sits between authenticated callers and the model, and what belongs somewhere else
The LLM inference gateway is the identity-aware policy enforcement point between authenticated users or agents and any model endpoint. It is the layer where authorization, data classification, and audit-record production live. This piece defines the term, walks through the four fields the gateway resolves per request, contrasts it with the inference server, model router, and API gateway it is often confused with, and shows why the audit-write path must be isolated from the caller. Applies to any deployment running an OpenAI-compatible or provider-native LLM API in production.
- Securing the Inference Lifecycle: The Five Stages Where the Enforcement Layer Has To Sit
The AI inference lifecycle is the sequence the application runs every time the model produces a response. Most security programs cover model training and the post-deployment monitoring stages but leave the inference path itself uninstrumented. This piece walks through the five stages of the inference lifecycle, the control points each stage exposes at the request boundary, the per-decision audit record the deployment has to commit, and the architectural pattern that closes the inference-time gaps a 2022-era AppSec program leaves open.