Why does the inference lifecycle need an inspection layer when the application has audit logs?

The application's audit logs describe what the application did. The inspection layer's audit records describe what the model decided and what the policy allowed. The two are different kinds of evidence. A regulator who asks "which policy applied to this request and what data did the prompt expose to the model?" reads the inspection layer's records because the application's logs do not have the policy or the data classification.

How does the inspection layer handle streaming responses from the model?

The inspection layer reads the response as the stream arrives. The classifier runs incrementally on the streamed tokens. The policy decision can interrupt the stream when the response triggers a high-severity rule (a secret appears in the response, the model proposes a tool call the policy blocks). The audit record captures the truncation event and the policy reason.

What does the inspection layer do when the application uses multiple models in a single workflow?

The layer runs in front of each model endpoint. The audit records carry the model identifier so a downstream consumer can reconstruct the multi-model workflow from the records. The policy evaluates per route, which means the policy can apply different rules to different models in the same workflow.

How does the inspection layer handle the system prompt that the application supplies?

The layer reads the system prompt as part of the assembled prompt. The system prompt may contain sensitive instructions the deployment does not want the model provider to inspect. The layer stores the system prompt hash in the audit record rather than the prompt itself when retention rules require it. The hash is recoverable through the application-side store the record references.

What about latency the inspection layer adds to the inference path?

The layer adds under 50 ms to each request. LLM inference takes 500 ms to 5 seconds. The inspection overhead is invisible against the model's response time. The classifier and the policy decision point run in parallel where possible, which keeps the layer's contribution to the request path below 50 ms.

Securing the Inference Lifecycle: The Five Stages Where the Enforcement Layer Has To Sit

The AI inference lifecycle is the sequence the application runs every time the model produces a response. The sequence starts when the application accepts the user's input and ends when the application commits the model's output to the next step in the workflow. The 2022-era AppSec program covered the moments before the user input arrives (authentication, authorization, input validation) and after the output ships (output encoding, audit logs of the application's actions). The five stages in between were uninstrumented because the model was a black box the application called with no inspection layer. The black-box assumption breaks in 2026 because the inference path produces decisions a regulator audits, an attacker targets, and a compliance auditor verifies.

I want to walk through the five stages of the inference lifecycle, the control points each stage exposes at the request boundary, the per-decision audit record the deployment has to commit, and the architectural pattern that closes the gaps the 2022-era AppSec program leaves open.

Stage 1: Input ingestion and identity binding

The first stage is the moment the application receives the user input and binds the request to a verified identity. The application validates the inbound session, resolves the user's directory record, and stages the prompt for the next stage.

The control point at this stage is the identity binding. The user the application authenticated has to be the user the inspection layer reads. The identity propagation through the application's session into the model call is the upstream obligation. A common failure mode is the application that authenticates the user at the front door but then issues the model call with the application's own service account, which severs the identity chain the audit record needs.

The audit record captured at this stage carries the verified subject identifier, the verification method (OIDC token, SAML assertion, mTLS certificate), the role memberships the directory holds at the moment of the request, and the session attributes that affect the policy (tenant, region, clearance level, BAA-coverage flag). The record is the foundation every subsequent stage builds on.

Stage 2: Prompt classification and context assembly

The second stage is the moment the application assembles the prompt that goes to the model. The prompt carries the user input, the system prompt the application supplied, the retrieved context the RAG pipeline pulled in, and the tool schema the application surfaces.

The control point is the classifier the inspection layer runs against the assembled prompt. The classifier evaluates the prompt content for PII, PHI, secrets, MNPI, regulated identifiers, and prompt-injection signals. The classifier evaluates the retrieved context for the same categories because the retrieved context is part of the prompt the model reads. The classifier returns a verdict the policy decision point uses at the next stage.

The audit record carries the classifier's verdict (each detected category with a confidence score), the system prompt hash (the system prompt may contain sensitive instructions), the retrieved context summary (a hash of each retrieved document), and the tool schema the prompt exposed. The record produces the evidence the regulator's question "what data did this request expose to the model?" reads against.

Stage 3: Policy decision and outcome commit

The third stage is the moment the policy decision point reads the identity context, the classifier verdict, and the routing inputs, and returns a per-decision verdict. The decision passes, blocks, modifies, or routes the request based on the rule set the deployment maintains.

The control point is the deterministic evaluation. A rule that triggers on a PHI classifier verdict for a user without PHI authorization returns a block. A rule that triggers on a secret classifier verdict returns a modify with the secret redacted. A rule that triggers on a high-severity prompt-injection verdict returns a block with a high-priority audit event the security team's SIEM consumes.

The audit record carries the policy version hash (the deployment maintains policy versions and the record captures which version evaluated the request), the rule identifier that produced the outcome, the decision (pass, block, modify, route), and the reason code. The record makes the decision reproducible. A regulator who reads the record can replay the same inputs against the same policy version and confirm the decision.

Stage 4: Model invocation and response inspection

The fourth stage is the moment the request crosses the HTTP boundary to the upstream LLM and the response returns. The application's network stack issues the request, the model produces the response, and the response returns to the application.

The control point is the inline inspection on both directions. The outbound inspection covers the prompt the policy decision already cleared, the destination endpoint, and the routing decision. The inbound inspection covers the response content (the response may contain PHI the model invented, code the model suggested, or tool calls the model proposed). The inbound classifier evaluates the response for the same categories the outbound classifier evaluated for the prompt.

The audit record carries the destination model identifier and version, the prompt cryptographic hash (the prompt's exact content is recoverable from the application-side store the record references), the response cryptographic hash, the response classification verdict, the latency from request to response, and the policy version hash for the response evaluation.

Stage 5: Output integration and action commit

The fifth stage is the moment the application applies the model's response to the next step in the workflow. The next step might be a database write, a tool call, an API request, a UI render, or a downstream service call.

The control point is the action authorization. A model response that proposes a tool call has to clear the policy that governs which tools the application's identity is authorized to call. A model response that produces text the application displays to the user has to clear the output policy that governs which classifications the application is authorized to surface. A model response that produces structured output the application persists has to clear the persistence policy.

The audit record carries the action the application committed (the database write, the tool call, the API request), the input that produced the action (the model response hash), the authorization the action ran under, and the outcome of the action. The record closes the loop from the user input to the side effect.

The architectural pattern that covers all five stages

The pattern places an inspection layer inline between the application and each HTTP boundary the inference path crosses. The layer reads the identity context the application propagates, the prompt content, the classifier verdict, the policy decision, the model identifier, the response, and the action authorization. The layer commits a per-decision audit record at each stage with the schema the regulator and the customer auditor consume.

The layer is the same component across the five stages because the architectural property (read the HTTP boundary, evaluate against identity-aware policy, commit the record) is the same. The deployment runs the layer in front of the LLM endpoint, in front of the tool endpoints, and inside the application's request handler where the input ingestion happens.

The pattern is decoupled from the model. A deployment that runs OpenAI for chat and Anthropic for long context routes both through the same layer with the same policy. A deployment that adds Bedrock for SOC 2 boundaries adds the route without changing the policy authoring surface. The audit records are interoperable across models because the schema is fixed.

Where the 2022-era AppSec program leaves gaps

The 2022 program covered the authentication and authorization at the application's front door. The five stages above run after the front door. The application sees the user as authenticated and proceeds with no additional inspection on each request the model produces a response for.

The 2022 program covered the input validation on the structured fields the user submitted. The five stages above include input the application did not validate because the input is the natural-language prompt the user typed into a chat box. The validation that worked on JSON fields does not transfer to the prompt.

The 2022 program covered the output encoding on the HTML the application rendered. The five stages above include output the model produced, which includes proposed tool calls and structured data the application persists. The encoding step happens after the action the model proposed has already executed.

The 2022 program covered the audit logs of the application's actions. The five stages above produce decisions the application's records cannot describe because the application was not the decision-maker. The self-attestation problem makes the application's records insufficient as audit evidence.

DeepInspect

This is the gap DeepInspect closes for the inference lifecycle. DeepInspect sits inline between the application and each HTTP boundary the inference path crosses. The inspection layer reads the prompt, the retrieved context, the response, the tool call, and the identity the application propagates. The layer evaluates identity-aware policy and commits per-decision audit records to durable, append-only storage with a cryptographic integrity signature.

The record series carries the verified subject identifier, the classifier verdict on the prompt and the response, the policy version hash, the rule identifier that produced the outcome, the model identifier and version, the action authorization, and the integrity metadata. The series satisfies EU AI Act Article 12, DORA Article 19, HIPAA 45 CFR 164.312, NIST AI RMF MANAGE 1.3, and ISO 42001 record-keeping obligations from a single pipeline. Book a technical deep dive at deepinspect.ai.