← Blog

Smolagents Audit Logs: Build an Evidence Record Around OpenTelemetry Traces

Parminder Singh
Parminder Singh··6 min read
Summarize with AI

Smolagents can export agent-run telemetry through OpenTelemetry, including run spans and token usage alongside inputs and outputs. An audit-ready deployment still needs verified caller identity, policy decisions, tool authorization, retention rules, and evidence that the final LLM request crossed the approved HTTP route.

Platform & Architectureai-securityagentic-aiauditforensic-auditidentity-and-authorization
Smolagents Audit Logs: Build an Evidence Record Around OpenTelemetry Traces

Smolagents can instrument an agent run with OpenTelemetry and send the resulting traces to Phoenix or another compatible backend, including MLflow and Langfuse. The framework records the chain of model interactions and tool steps that made a run possible. That trace helps an engineer debug an agent. An audit record has a harder job: prove who initiated the run, which policy applied, what crossed the model boundary, and which action was authorized at each step.

I would keep both records and refuse to pretend they are interchangeable.

TL;DR

  • Smolagents OpenTelemetry traces show run steps and model interactions. They also record inputs, outputs, and token usage, which makes them useful engineering evidence.
  • An audit-ready record must add verified caller identity, policy version, route authorization, tool decision, retention state, and a link between each LLM request and its parent run.
  • DeepInspect can record policy decisions for authenticated LLM HTTP traffic routed through it; local code and tool execution remain separate controls, as does trace-backend custody.

Smolagents records a multi-step run

The Smolagents telemetry tutorial describes OpenTelemetry instrumentation for production inspection and monitoring. Its MLflow example tracks runs and spans, with inputs, outputs, and token usage. Phoenix and Langfuse examples expose the sequence of model and tool activity in trace views.

That sequence matters because a multi-step agent rarely maps to one model call. A run may ask a model to plan and invoke a search tool. It can then pass the observation into another prompt and call a managed agent before producing a final answer. The Smolagents agent reference describes each step as a thought followed by one tool call and its execution.

A useful trace preserves that tree. The root span identifies the run. Child spans carry model and tool activity. Timing and status reveal where the run stalled or recovered.

A trace starts with observability

OpenTelemetry answers operational questions well. It shows the order and duration of spans. It also shows which component returned an error and how inputs became outputs across a run. Those fields support debugging and performance analysis.

An auditor asks a different set of questions. Which authenticated employee or workload started this run? Which policy version approved the model route? Was the tool call permitted for that identity? Did the prompt carry payroll data? Which record proves the response passed through the same control point?

The answer cannot be inferred from a span name such as agent.step.4. Add those fields deliberately. The record for a production Smolagents run should bind the root trace identifier to the caller and tenant. It should also record the purpose, agent version, model route, and policy decision.

This is the distinction I use in AI agent observability: traces explain behavior, while independent policy records prove authorization.

The evidence schema needs seven fields

A release test should verify that every production run produces these fields:

  • Caller identity: the authenticated identity that initiated the run, recorded as a user or as a non-human workload or agent.
  • Run identity: a stable trace identifier shared by the root span and every child step.
  • Agent version: the deployed Smolagents package, agent configuration, tool list, and prompt revision.
  • Route decision: the model endpoint and requested model, plus the policy version and permit or deny result.
  • Data decision: the classification applied to the prompt, followed by any redaction or block action.
  • Tool decision: the requested tool, arguments digest, authorizing identity, and outcome.
  • Custody state: the telemetry destination and retention period, plus the access role and deletion path.

Use a visible test marker such as AUDIT-CASE-417 in a synthetic prompt. The marker should appear in the approved trace fields and the corresponding request-boundary record without being copied into unrelated logs. A reviewer then has a physical thread to follow through the run instead of matching timestamps by eye.

Tool execution has a separate owner

Smolagents supports CodeAgent, which writes tool calls as Python, and ToolCallingAgent, which emits structured tool calls. The framework's secure code execution guidance warns that local execution of model-generated code can damage files and cloud resources. API services and network security can also be affected. It recommends stronger isolation options as risk increases.

Those actions sit outside the LLM HTTP request once execution begins. A model-call record can show the response that proposed a tool action. It cannot prove that the local interpreter or sandbox enforced the right permission. The same gap applies to a database or external API.

The audit chain therefore needs two linked decisions. The request-boundary record covers the prompt and model response. A tool-execution record covers the actual action and target, together with the authorization result and side effect. The parent trace identifier connects them without assigning one control responsibility it cannot carry.

Agentic AI audit trails require this parent-child structure because a flat list erases causality.

Telemetry can contain the data under review

Inputs and outputs are useful during an investigation, yet they may contain PII or source code. Credentials and customer records may also appear. Sending every span to a hosted backend can duplicate the data channel the audit process is meant to supervise.

Define field-level rules before enabling collection. Keep the route and identity, along with the classification and decision. Retain the timing and digest needed for evidence. Redact or omit raw prompt content when the investigation value fails to justify retention. Restrict trace access by role and test deletion against the documented retention period.

A practical test uses AUDIT-CASE-417 plus two synthetic values: a customer number and a fake API key pattern. The collector should preserve the case marker and decision metadata while removing the fake secret. Open the stored trace after export and inspect what actually arrived. Configuration screenshots cannot substitute for that check.

The release test follows one run

Create a synthetic identity named claims-review-agent-12 and allow it to call one model route plus one read-only tool. Then run four cases:

  • The approved identity calls the approved model route and read-only tool.
  • The same identity requests a forbidden model route.
  • The identity asks for a write-capable tool outside its role.
  • A request reaches the model endpoint without the required gateway path.

For each case, retain the root trace identifier and child spans. Keep the request-boundary decision, tool decision, and expected result with them. The denied model request should stop before inference. The denied tool action should leave the target unchanged. The bypass attempt should fail at the network or service boundary.

Repeat the test after upgrading Smolagents or changing the instrumentation package. The API reference labels Smolagents experimental and subject to change, which makes versioned evidence part of the release process rather than annual paperwork.

Independent records survive trace failure

An OpenTelemetry exporter can drop spans when the collector is unavailable or a queue fills. Expired credentials can cause the same failure. Decide how the agent behaves before production traffic arrives. A fail-open telemetry path may be acceptable for debugging data. The independent authorization record for a regulated model request needs a stronger guarantee.

The acceptance run should stop the collector, send one approved request, and verify the chosen behavior. Record exporter retries and queue limits. Include dropped-span counters and the point where the system alerts an operator. Then confirm that the request-boundary policy decision still exists in its system of record.

OpenAI Agents SDK audit logs face the same write-path problem. Framework traces are valuable. Evidence independence comes from a control outside the framework's own write path.

DeepInspect

DeepInspect can enforce policy on authenticated HTTP requests sent by a Smolagents application to an LLM endpoint when that route is deliberately placed behind the proxy. It can bind the supplied caller identity to model and data policy, then produce a per-decision audit record linked to the Smolagents trace identifier.

Local Python execution and tool side effects remain under the application and platform owners. The same applies to agent memory, OpenTelemetry export, and trace-backend retention. Linking those records creates a complete chain without stretching the HTTP control boundary past the traffic it can inspect.

Book a technical deep dive at deepinspect.ai.