LLM observability
LLM observability is the collection and correlation of traces, token counts, latency percentiles, cost attribution, and outcome metadata for every LLM call an organization makes. The telemetry answers the operational questions: which routes run slow, which teams drive spend, which prompts produce refusals or errors, and how completion behavior shifts after a model version change.
What the telemetry captures
A complete trace spans the full request lifecycle: gateway receive time, classification verdict, model call duration, token counts in and out, the completion outcome, and the policy decision attached to the request. Cost attribution aggregates those token counts per team, per route, and per model, which turns a single monthly provider invoice into a line-item budget conversation. Version-pinned baselines matter most: when a provider ships a new model revision, the observability layer is where a latency regression or a refusal-rate jump becomes visible before users report it.
Where observability stops
An observability trace describes traffic the model has already answered. That record carries real forensic and capacity-planning value, and zero prevention capability at machine speed: Google Mandiant's M-Trends 2026 report measured the median attacker handoff at 22 seconds, a window that closes long before a dashboard alert is read. Prevention lives earlier in the path, at the inline policy decision that runs before the model sees the request. Mature deployments wire the two layers together, with the enforcement point emitting per-decision audit records and the observability stack consuming them for dashboards, alerts, and spend reports.
Related reading
- AI Agent Observability: The Signals That Turn Autonomous Behavior From a Black Box Into a Debuggable System
Application observability (metrics, logs, traces) misses the signals that matter for AI agents: which tools the agent called, which sub-agent it delegated to, which policy decision permitted or denied each step, how many tokens each decision cost. Autonomous behavior without per-step observability is an unauditable black box. This covers the signals a production agent has to emit, the OpenTelemetry semantic conventions taking shape, and where the AI request boundary fits in the telemetry pipeline.
- LLM Audit Logging: The Implementation Pattern That Holds Up Under Regulator Review
LLM audit logging implementations split along three architectural patterns: in-application logs, sidecar collectors, and inline inspection layers. The inline pattern is the only one that produces records the EU AI Act Article 12, DORA Article 19, and Fannie Mae LL-2026-04 reviewers accept because it is the only one that satisfies the write-path independence test. This piece walks through the three patterns, the architectural reason the first two fall short, the integration points the inline pattern requires, the field set the records have to carry, and the latency budget that fits a production deployment.
- AI Audit Logs: The Format Spec That Survives EU AI Act, DORA, and Fannie Mae Review
AI audit logs that survive regulatory review carry a specific set of fields the EU AI Act Article 12, DORA Article 19, Fannie Mae LL-2026-04, NIST AI RMF, and HIPAA all expect on the same record. The fields cover identity, decision provenance, model identity, policy state, and integrity metadata. The format has to support per-record retrieval and per-series replay. The write path has to sit outside the application so the application cannot modify the record. This piece walks through the field-level format specification, the integrity model, the storage characteristics, and the deployment pattern that produces records the regulator and the customer auditor will accept.