LlamaIndex Audit Logs: What the Instrumentation Module Records and What LlamaCloud Adds

A production RAG stack built on LlamaIndex produces two streams of records, and they come from different systems with different purposes. The open-source framework emits Spans and Events through an instrumentation dispatcher, which is developer telemetry. LlamaCloud, the managed parsing and indexing service, keeps organization-level audit logs on Enterprise plans, which is administrative history. I want to walk through what each one actually captures, because the distinction decides what a security team can reconstruct after an incident.

TL;DR

• The LlamaIndex instrumentation module emits Span and Event objects (LLMChatStartEvent, LLMChatInProgressEvent, LLMChatEndEvent) to registered handlers, carrying model details, messages, and responses.
• Those events carry no authenticated user, role, or authorization context. The framework records what the code did, not who asked for it.
• LlamaCloud lists audit logs and SSO as Enterprise-tier features, covering organization and project administration through its roles and membership APIs.
• Neither record answers the question a regulator asks: which authenticated person caused this document to be retrieved and sent to this model at this moment.

What the instrumentation module emits

LlamaIndex's instrumentation module is built around four pieces. An Event represents "a single moment in time that a certain occurrence took place within the execution of the application's code." A Span represents "the execution flow of a particular part in the application's code and thus contains Event's." An EventHandler listens and runs custom logic. A Dispatcher "emits Event's as well as signals to enter/exit/drop a Span to the appropriate handlers."

The LLM interaction events are the ones a security engineer cares about. LLMChatStartEvent, LLMChatInProgressEvent, and LLMChatEndEvent carry model details, the messages sent, the response returned, and streaming deltas. That is a genuinely useful record for debugging a retrieval chain that returned the wrong chunk, and it is the same data the observability integrations (Arize Phoenix, OpenTelemetry exporters, Dynatrace) consume downstream.

Picture the artifact a security team holds at 2am during an incident review: a span tree, a model name, a token count, and the full text of a prompt with no name attached to it. Every field in that record describes the program. None of them describes the person.

The dispatcher has no concept of a caller

The instrumentation design decision here is deliberate and correct for its purpose. A dispatcher sits inside the Python process, attached to the call stack, and it fires when chat() is invoked. Whatever authenticated identity existed at the edge of the application, the session cookie, the OIDC token, the service account, was resolved and discarded several layers above the point where the dispatcher runs.

Adding a user_id field to a custom EventHandler is possible and teams do it. That field is application-asserted, which means it carries exactly as much weight as the application's own correctness. A bug in session handling, a mis-scoped background job, or an agent looping on behalf of a user who logged out three hours ago all produce a user_id the log will happily record. The same limitation shows up in framework-level records across the category, which I covered in why the LangSmith user ID field is not an identity check.

Calling a dispatcher-emitted span an audit record is a category error, and I think the way most observability vendors market these integrations actively encourages it.

What LlamaCloud adds on Enterprise

LlamaCloud is a separate product from the open-source framework: a managed service for document parsing, extraction, indexing, and retrieval. Its Enterprise tier documents SSO integration and audit logs alongside SOC 2 compliance and on-premises deployment options.

The administrative surface those logs cover is visible in the LlamaCloud API, which exposes organizations, projects, memberships, role assignment, per-user project access, and invites. Self-hosted LlamaCloud deployments support OIDC against Microsoft Entra ID and Okta, so the identity plumbing genuinely exists at the control-plane layer.

That control plane is where the audit log lives. It records that an administrator granted a user access to a project on a Tuesday. It does not record the forty thousand retrieval calls that user's API key made against that project on Wednesday, because those calls travel a different path entirely, straight to the data plane.

The gap sits between the two records

Trace a single request through a LlamaCloud-backed RAG application and the split is obvious. A user authenticates to the application. The application resolves their identity, then calls LlamaCloud with a project-scoped API key that belongs to the application, not to the user. LlamaCloud returns the matching chunks, the framework's instrumentation dispatcher emits an LLMChatStartEvent carrying the assembled prompt, and the model responds.

Three systems recorded something along that path. The IdP recorded a login. LlamaCloud recorded nothing about this request in its audit log, because the audit log covers administration. The dispatcher recorded a prompt and a model name with no actor. Nowhere in that chain does a single record bind an authenticated human to a specific document retrieval and a specific model call, which is precisely the binding that EU AI Act Article 12 logging obligations and comparable regimes are written around.

This is the same architectural shape described in the post-authentication gap: identity is checked once at the front door and never consulted again on the traffic that follows. The security patterns worth applying around a LlamaIndex deployment start from that observation rather than from the framework's own telemetry.

What an identity-bound record has to contain

For a retrieval-augmented application to produce something a compliance reviewer accepts, each record needs four fields the current stack cannot supply from inside itself: the authenticated principal as asserted by the identity provider rather than by application code, the role or group membership that principal held at request time, the classification of the data that entered the prompt, and the policy decision that permitted the call with the policy version that produced it.

Those four fields have to be captured by something that sits on the request path and independently verifies identity, rather than by a library that runs after identity has already been thrown away.

DeepInspect

This is the gap DeepInspect closes. DeepInspect sits inline as a stateless proxy in front of HTTP-based LLM endpoints, so every call a LlamaIndex application makes to a model, whether that is OpenAI, Anthropic, Bedrock, or a self-hosted server, passes through a point that verifies the caller's identity against the identity provider before the request continues.

Because the check happens on the request path rather than inside the application process, the resulting record carries the authenticated principal, the role, the data classification of the prompt content, and the policy decision that allowed or blocked the call. LlamaCloud's audit log keeps doing what it does well, tracking who was granted access to which project. The instrumentation dispatcher keeps producing the span trees that make a broken retrieval chain debuggable. DeepInspect supplies the third record, the one that ties a named person to a specific inference request. Book a technical deep dive at deepinspect.ai.