← Blog

DSPy Audit Logs: Build an Evidence Map for Every LM Request

Parminder Singh
Parminder Singh··6 min read
Summarize with AI

DSPy exposes LM history, MLflow traces, callbacks, cache behavior, and provider records at different layers. This guide maps each source to the audit question it can answer, identifies the identifiers needed to correlate a production prediction, and defines the independent policy record required for deliberately routed HTTP model calls.

Platform & Architectureai-securityauditforensic-auditobservabilitypolicy-enforcement
DSPy Audit Logs: Build an Evidence Map for Every LM Request

A DSPy prediction can leave evidence in five places before an investigator opens a case. inspect_history() shows LM invocations. MLflow can trace the prediction tree. Callbacks can emit selected lifecycle events. The model provider records its own request. A routed policy point can record the decision made before the model call. Calling any one of these the DSPy audit log creates gaps in collection and interpretation.

I would start with an evidence map, not a logging toggle. Each record needs a named question and a stable join key, plus a retention owner and a test that proves the record exists after a real prediction.

TL;DR

  • inspect_history() is a local LM debugging view; it omits retrievers and tools, along with call relationships, parameters, and latency.
  • MLflow traces can show workflow steps, while DSPy callbacks emit selected module, LM, adapter, and tool events, plus evaluation and optimizer events.
  • Provider telemetry answers a separate delivery question and needs correlation to the DSPy prediction.
  • Deliberately routed HTTP LM calls need an independent policy decision record tied to the authenticated caller.

inspect_history() is a debugging view

DSPy's observability tutorial describes inspect_history() as a utility that prints LM invocations made so far. That is useful when a response looks wrong and an engineer wants to see the messages and model output. Its collection boundary is narrow. The same tutorial says it logs only LM calls and arranges multiple calls in a monolithic history. It omits parameters, latency, component relationships, retrievers, tools, and custom modules.

Treat the output as local debugging evidence with an explicit capture procedure. Record the process and environment, then record the collection time and number of entries requested. Preserve the relevant invocation instead of a cropped terminal screenshot. The physical detail matters here: a developer scrolling through green and white terminal text at 11:40 p.m. can spot the bad prompt, but an auditor still needs to know which production prediction produced that screen.

This is one slice of the wider AI agent observability problem. History can explain one LM exchange. It cannot establish the complete execution path around that exchange.

MLflow traces reconstruct the prediction tree

DSPy's observability tutorial shows mlflow.dspy.autolog() creating a trace for each prediction. The trace UI exposes inputs and outputs for workflow steps, including a retriever result and an LM's input and output, along with its configuration. That execution tree addresses several limitations of the flat history view. It can show which step supplied stale context and where the LM call sat in the prediction. It also shows what the surrounding component returned.

A trace still needs an acceptance test. Trigger a prediction with a distinctive case identifier, retrieve the resulting trace, and confirm that the expected workflow steps appear. Then export the trace through the approved evidence path. A browser tab proves that the trace UI rendered at collection time. The retained trace record supports later review.

Trace content can include prompts, retrieved text, and model output. Define who can read it and how long it remains available. Identify which fields require redaction before an external exporter receives them. MLflow is the execution evidence owner in this map. It remains application telemetry rather than an independent network policy record.

Callbacks create selective lifecycle evidence

DSPy's observability documentation lists callback handlers around modules and LM calls. Separate handlers cover adapter formatting, tool activity, evaluation runs, and optimizer activity. The LM API reference also accepts callbacks that run before and after each model request. Those hooks support a purpose-built event stream with the identifiers and fields your incident procedure needs.

Selection creates responsibility. A callback records the events its implementation handles and the fields it emits. Review the registration point and failure behavior, plus the output destination and treatment of exceptions. Run one successful prediction and one provider failure. Confirm that start and end events join to the same prediction. A failed call should produce an intelligible terminal state.

Keep callback scope visible in the evidence map. Module and tool callbacks describe local DSPy execution. Evaluation and optimizer callbacks describe development or optimization activity. LM callbacks sit closest to the outbound request, but they remain inside the calling application. The map should never ask an application callback to prove that an external policy system independently allowed the request.

Provider telemetry proves a separate delivery event

dspy.LM selects a model and API family, as well as an engine. The current API reference says the default auto behavior can use native provider APIs, while explicit settings can select chat, responses, text, or the LiteLLM compatibility engine. Custom engines own their protocol and connection settings. One DSPy application can therefore produce requests through materially different inference routes.

Provider telemetry answers whether the provider received and processed a request under a particular account or project. Capture its provider request identifier, model, timestamp, and status. Also capture token usage where available and the account context. Then join that record to the LM callback or trace. A model name alone is a weak key because concurrent calls can share the same model and second.

Retries complicate correlation between application and provider records. DSPy's LM reference documents three retries by default for transient errors, using exponential backoff. One logical LM operation can generate more than one transport attempt. Keep a stable logical call identifier plus an attempt number. This prevents an investigator from mistaking a retry sequence for several independent user actions.

The evidence map assigns one question to each source

A useful DSPy evidence map should be short enough to use during an incident. I would put these entries in it:

  • LM history: What messages and response did the local DSPy process retain for the selected LM invocation?
  • MLflow trace: Which modules and workflow steps contributed to the prediction? How were their inputs and outputs connected?
  • DSPy callback stream: Which configured lifecycle events fired? Which local prediction and call identifiers did they carry?
  • Provider telemetry: Which inference attempt reached the selected provider account and model through the recorded protocol route?
  • Independent request-policy record: Which authenticated principal requested the routed LM call? Which destination and policy version were evaluated, and what decision was made?

For each entry, name the system of record and event owner. Include the join keys and clock source, plus the retention period, access group, and known blind spots. The result follows the same principle as signed audit logs for AI requests: evidence gains value when the record captures identity and destination, decision and policy state, plus time and integrity context at the point that made the decision.

Correlation needs a tested identifier chain

Choose a prediction identifier at the application boundary and propagate it into every supported evidence source. Add a separate LM call identifier for each model operation and an attempt identifier for retries. Where the provider accepts metadata or returns a request ID, preserve that value in the callback or trace. For routed HTTP calls, pass the approved correlation value to the policy layer and return its decision identifier to application telemetry.

The acceptance test should use one prediction that invokes the LM twice, with the second operation forced through one transient failure. Retrieve the DSPy trace and callback events, plus the provider and policy records. The expected package contains one prediction and two logical LM calls. It also contains three transport attempts. Every timestamp should fall within the documented clock tolerance.

This chain also exposes the post-authentication gap. A user ID attached at the web tier can disappear before the LM request. Test that propagation with two users plus one service account. Shared provider credentials should still produce distinct caller context at the application and policy layers.

DeepInspect

DeepInspect covers authenticated HTTP LM invocations that a DSPy application deliberately routes through it. The application supplies caller and correlation context. DeepInspect evaluates the request destination and prompt classification against versioned policy before forwarding an allowed call. It then records the resulting decision identifier for correlation.

DSPy modules, retrievers, tools, optimizers, local traces, caches, and whole-program artifacts remain outside that boundary. MLflow and callbacks retain ownership of local execution evidence. Provider telemetry retains ownership of provider receipt. Direct LM calls that bypass the configured HTTP route also sit outside the DeepInspect record. DSPy explains the prediction and the provider confirms inference attempts. DeepInspect records policy decisions on the routed call.

Book a technical deep dive at deepinspect.ai.

Frequently asked questions

Does DSPy have one audit log?

Each DSPy evidence source answers a different audit question. inspect_history() covers LM invocations retained by the process. MLflow tracing can capture a prediction's workflow steps. Callbacks can emit selected lifecycle events. Provider records describe inference attempts that reached the provider. A complete collection plan names each source and correlates them with prediction and logical call identifiers, plus attempt, provider request, and policy decision identifiers.

Can `inspect_history()` replace tracing?

inspect_history() records LM calls while omitting retrievers, tools, and custom modules, according to DSPy's documentation. Parameters, latency, and relationships among components also sit outside that record. Tracing can preserve the wider prediction structure. Use history for focused debugging and a trace when the investigation needs to reconstruct how several workflow steps produced the model request.

Do LM callbacks prove the provider received the request?

A callback proves that configured application code observed a lifecycle event. Provider telemetry proves receipt and processing on the provider side. Network failures and retries can create differences between those records. Callback exceptions and custom engine behavior can do the same. Join both sources with a logical call identifier and the provider's request ID where the route exposes one.

What should a DSPy audit acceptance test contain?

Use a known prediction with two LM calls and one forced transient failure. Retrieve the trace and callback stream, plus the provider attempts and routed policy decisions. Confirm the expected counts and identifiers. Then confirm caller context, model destination, policy version, decision, and timestamps. Preserve the raw records and the query used to collect them.