LLM Observability: What Traces and Token Metrics Tell You, and the Enforcement Record They Never Produce
LLM observability instruments the request path with traces, spans, token counts, latency, cost, and quality evaluations, so an engineering team can debug a slow chain or a regression. That telemetry is forensic. It tells you what happened after it happened. At an attack tempo measured in seconds it does not prevent anything, and the records it writes carry model and latency but not the caller identity, data classification, and policy decision an auditor asks for. This piece separates the debugging value of observability from the enforcement and audit layer it cannot replace.
An LLM observability pipeline wraps each model call in a trace, records the prompt, the completion, the token counts, the latency of every span, and often a downstream quality score, then ships all of it to a backend where an engineer can replay a slow or wrong chain. That instrumentation earns its place for debugging and cost control. It records what happened, in detail, after the model already returned. Two things follow from that timing, and both matter for security. Observability cannot prevent a bad request at machine speed, and the record it produces carries model and cost but not the identity, data classification, and policy decision an auditor needs. I want to separate what observability is good for from the enforcement layer it cannot stand in for.
What LLM observability instruments
A modern observability setup gives a team real visibility into the request path, and running it is sound engineering.
Traces and spans. Each request becomes a trace, and every step, the retrieval, the prompt assembly, the model call, the tool invocation, becomes a span with its own timing. A slow chain shows exactly which span cost the seconds.
Token and cost accounting. Input and output token counts roll up per request, per feature, and per model, so a finance owner can see where spend concentrates.
Quality and eval signals. Offline and online evaluations score outputs for correctness, groundedness, or policy alignment, which catches regressions after a prompt or model change.
Prompt and response capture. The full prompt and completion are stored for replay, which is how an engineer reproduces a production issue.
Where observability stops being a control
Every one of those capabilities describes the past. The trace is written when the span completes, which is after the model saw the prompt. Google Mandiant's M-Trends 2026 report, based on 500,000+ hours of frontline incident response, found the median time between initial access and handoff to a secondary threat group collapsed to 22 seconds in 2025. Against that tempo, a control that reports what happened is a forensic tool and not a preventive one. A prompt carrying customer PII to an external model is already gone by the time the trace lands in the dashboard. Observability tells you the data left. It was never positioned to stop it leaving.
The record observability writes, and the record an auditor wants
The two records look similar and serve different masters. An observability span records model, latency, token count, and often the raw prompt, and it is written by the application's own instrumentation for the application's own team. An audit record has to answer who the authenticated caller was in corporate identity terms, what data classification the prompt carried, which policy applied, and what the system decided, and it has to be produced independently of the application under review. A span authenticated with a shared service key maps every user to one identity, which is the field a regulator cares about most and the field the span never held. Telemetry written by the system under audit is self-attestation, and self-attested logs fail the independence test that any serious audit applies.
What inline enforcement adds on the same traffic
Prevention requires a decision on the request before it reaches the model, not a trace after. A policy decision point sits inline on the path to the model, reads the caller identity, classifies the prompt, evaluates policy, and returns a verdict:
The block happens in 41 milliseconds, before the prompt reaches the provider, and the same evaluation writes the identity-bound record that satisfies the audit. Enforcement and audit come from one decision at the request boundary. Observability then does what it is good at, tracing the requests that were allowed to proceed. The layers complement each other, and I would keep both in any production stack. For the telemetry that lives on the gateway itself, see AI gateway observability; for the detection side, see LLM security monitoring.
Compliance framing
For regulated AI, the observability backend is not the audit trail. EU AI Act Article 12 requires automatic, traceable event logging for high-risk systems, including inputs and the natural persons involved, applying from August 2, 2026 per the European Commission. A trace store built for engineers to debug latency sits in a weaker position than an identity-bound record produced independently of the application. The distinction between forensic telemetry and enforceable, independent audit is the whole game, and conflating the two is how teams arrive at an audit with dashboards instead of evidence.
DeepInspect
This is where DeepInspect fits. DeepInspect is a stateless proxy that sits inline in front of your AI traffic and is model-agnostic, so one layer covers OpenAI, Anthropic, Bedrock, Vertex, Azure OpenAI, and self-hosted endpoints. For every request it evaluates the caller's identity and role, the data classification in the prompt, and organizational policy, then returns a pass or block decision before the traffic reaches the model. Enforcement adds under 50 milliseconds in internal testing, invisible against the 500 milliseconds to several seconds a model completion takes, and the default on ambiguity is deny.
For every decision it commits an identity-bound audit record before the response returns, written independently of the calling application. That record carries the identity, policy, data classification, and outcome your observability spans never held. DeepInspect does not replace your tracing, cost accounting, or eval pipeline. It adds the inline enforcement and independent audit layer that observability, by its after-the-fact nature, cannot provide. Book a demo today.
Frequently asked questions
- Is LLM observability a security control?
It is a forensic and debugging tool. Traces and token metrics record what happened after the model responded, so they support investigation and cost control. At machine-speed attack tempo they do not prevent a bad request, which requires a decision on the request before it reaches the model.
- Can I use my observability logs as an audit trail?
Usually not on their own. Observability spans are written by the application's own instrumentation, often lack corporate identity and data classification, and are self-attested. An audit under EU AI Act Article 12 wants an identity-bound record produced independently of the application under review.
- How is inline enforcement different from observability?
Enforcement evaluates the request and blocks or allows it before it reaches the model, in tens of milliseconds. Observability records the request after the model has already processed it. One prevents, the other explains, and a production stack benefits from both.
- Does enforcement replace my tracing pipeline?
No, enforcement decides which requests proceed and writes the audit record. Tracing then instruments the allowed requests for latency, cost, and quality. They operate at different points on the request path and cover different needs.
- What latency does an inline enforcement decision add?
The decision adds under 50 milliseconds in internal testing, against the 500 milliseconds to several seconds a model completion takes. The overhead sits inside the model's own latency and is not perceptible to the caller.