Is LLM observability the same as APM for AI?

It borrows the trace model from APM and adds signals APM has no concept of, principally prompt and completion payloads and evaluator-produced quality scores. The operational difference is that an APM trace describes a deterministic system, while an LLM trace describes a probabilistic one where the same input can produce different output on consecutive calls.

What does observability cost to run?

Payload storage dominates, since prompts and completions are far larger than conventional log lines. Most teams sample production traffic and keep full capture in development, which controls spend and creates the gap that matters when someone asks for a specific historical request.

Does self-hosting solve the prompt-privacy problem?

It addresses the transfer concern by keeping payloads inside your infrastructure. The trace store still holds whatever personal data those prompts carried, so it remains a processing location with retention, access control, and erasure obligations attached. Self-hosting changes where the data sits rather than removing the obligation.

Which instrumentation model should I choose?

Run both, for different purposes. SDK or framework instrumentation gives the semantic detail needed to debug quality, and proxy capture gives the coverage and independence needed for governance. They are complementary layers rather than competing options.

Do evaluator scores count as guardrails?

They measure rather than gate. A score describes output the user already received, generally on a sample. An enforceable guardrail sits on the request path, evaluates every call, and returns a deterministic permit or deny. Related detail in AI agent guardrails.

What Is LLM Observability: The Four Signals, the Instrumentation Models, and What It Cannot Tell You

A conventional web service fails loudly. The request returns a 500, the stack trace names a line, and the fix follows. An LLM-backed feature fails quietly: the response comes back with a 200 status, well-formed, confident, and wrong. Nothing in the HTTP layer registers a problem.

LLM observability exists to close that gap. It captures enough detail about each model call and each step around it to reconstruct why a specific request produced the output it did.

I want to define the four signals it collects, walk the three instrumentation models that determine how much of your traffic it actually sees, and then be direct about the two categories of question the practice structurally cannot answer.

The four signals

Conventional observability rests on metrics, logs, and traces. LLM work adds a fourth signal and changes the shape of the other three.

Traces capture the execution tree of a single request: prompt construction, retrieval steps, the model call, tool invocations, and any loop the agent ran. The nesting matters, because a bad answer usually originates several steps before the model call.

Payloads are the prompt and completion text. This signal has no equivalent in conventional observability, and it drives most of the value and all of the governance difficulty. The input field holds whatever the user typed, which means your trace store inherits the sensitivity of the source data.

Cost and token metrics track input tokens, output tokens, and spend, ideally attributable to a user, a feature, or a customer.

Quality scores come from evaluators run over captured traces: relevance, groundedness, toxicity, or a custom judge. Unlike the other three, this signal is produced rather than observed, usually on a sample, and usually after the response reached the user.

The three instrumentation models

Where the data comes from determines what fraction of your AI traffic you can see. This is the part most evaluations skip.

SDK instrumentation gives the richest detail, because the code knows its own semantics: which retrieval step ran, which prompt template, which branch. It covers exactly the applications someone remembered to instrument.

Framework hooks trade breadth for zero manual work inside a supported framework. Teams running several frameworks get partial coverage, and bespoke agents get none.

Proxy capture inverts the trade. Application internals are invisible, since the proxy sees an HTTP request rather than the retrieval step behind it. Coverage becomes a network property: reaching the provider means traversing the proxy, so the analyst's notebook script and the vendor tool calling a model on your data both appear.

Coverage differences here are large in practice. An organisation with three engineering teams, a data science group, and a handful of SaaS products with AI features embedded will instrument the applications and miss most of the rest.

What a trace looks like

A worked example makes the abstraction concrete. A support assistant answering a billing question produces roughly this:

Reading it top to bottom, the groundedness flag at 0.62 points backwards to the retrieval step, where the top chunk scored 0.81 against a query that omitted the customer's plan tier. The model behaved reasonably given a context window that lacked the relevant paragraph. Without the trace, this arrives as "the bot gave a wrong refund answer" and takes a day to reproduce.

The questions it answers well

Observability earns its place on four kinds of question.

Why did this specific response go wrong, answered by walking the trace back from the output to the step that introduced the error. Where is the latency, answered by the span durations, which usually indicate retrieval or a serial tool chain rather than the model. What does this feature cost per user or per customer, answered by token attribution. Whether quality is drifting after a prompt or model change, answered by scores over time.

Those four cover most of the daily work of running an AI feature, and a team without them is debugging by intuition.

The two questions it cannot answer

Both limits come from architecture rather than from any product's maturity.

Was this call permitted

Observability records what already happened. Authorization decides what may happen, and deciding requires a position on the request path with the authority to refuse.

An evaluator scoring a prompt for injection patterns, on a sample, after the response returned, produces a finding rather than a prevention. Mandiant's M-Trends 2026, drawn from more than 500,000 hours of incident response, put the median handoff from initial access to a secondary threat group at 22 seconds, down from over eight hours in 2022 (Help Net Security). A control that reports at that tempo is documentation.

The distinction becomes concrete with a specific case. An engineer pastes a customer contract into a prompt. Observability captures it, flags PII if evaluators are configured, and the contract is already at the provider and now also sits in the trace store. A policy layer classifies the prompt before it leaves and refuses the call if this role may not send that data class to that model.

Can this stand as evidence

Audit evidence has requirements that telemetry does not meet, and two of them conflict directly with how observability is built.

Sampling is the first of them. Telemetry is priced on the assumption that a representative subset answers the question, which holds for latency and fails for an auditor asking about one request on one date. Retention compounds it, since observability windows get trimmed whenever the bill grows while high-risk AI evidence obligations run to years.

Self-attestation is the second and the sharper one. SDK-emitted records are written by the application being audited. That arrangement invites selective logging, allows suppression by the system that failed, and loses the record entirely when a process crashes after the model responded but before the span flushed. EU AI Act Article 12 requires automatic recording of events across a high-risk system's lifetime, including timestamps, input data, and identification of the natural persons involved (Practical AI Act), with high-risk obligations applying from August 2, 2026. A store the application controls answers that requirement weakly.

How the two layers fit together

Running both is the normal end state, and the split is clean. An observability platform owns quality, latency, and cost, instrumented in the application where the semantic detail lives. A policy layer on the wire owns authorization and evidence, covering every caller and writing unsampled records the application cannot alter.

Teams that try to make one layer do both jobs generally end up with an observability tool holding compliance evidence in a system priced to delete it, which surfaces during the first audit. Related reading in AI gateway observability and LLM observability tools.

DeepInspect

This is exactly what DeepInspect does. DeepInspect sits inline between your users or agents and the LLM APIs they call, evaluating identity, data classification, model authorization, and organizational policy for every request, then making a pass or block decision before traffic reaches the model.

Capture happens on the wire, so coverage includes callers no one instrumented. Enforcement runs inline with overhead under 50 ms in internal testing, set against inference times of 500 ms to several seconds, which puts the decision below the noise floor of the model's own latency. Every decision commits an unsampled structured record with the authenticated identity, the resolved role, the data classes detected in the prompt, the policy version in force, and the outcome, written before the response returns on storage the calling application has no access to. Book a demo today.