Implement OpenTelemetry GenAI Conventions at an AI Gateway

An AI gateway observes one logical model operation while performing several local actions: receive the request and bind identity; classify content and evaluate policy; call the provider, then inspect and return the response. OpenTelemetry's GenAI semantic conventions standardize the model-operation side of that trace. The gateway still needs its own policy span and durable decision record. Mixing those records produces telemetry that looks complete while losing the distinction between engineering diagnosis and security evidence.

TL;DR

• Use standard GenAI fields for operation and provider, requested and response model, server address and finish reason, plus low-cardinality error type.
• Create a separate gateway policy span for classification and enforcement instead of renaming a provider inference span.
• Treat prompt and response content as opt-in sensitive data, with explicit storage and access policy.
• Correlate traces with signed decision records, then test schema changes because many GenAI attributes still carry development status.

The standard span describes the caller's model operation

The current OpenTelemetry GenAI span specification says spans are logical operations as observed by the caller. A span starts when the operation begins and ends after the response is fully received or the operation terminates. Automatic retries belong inside that logical duration.

For a gateway, preserve the application trace context and create the model-operation span around the upstream call. The applicable standard value belongs in gen_ai.operation.name, while gen_ai.provider.name comes from the instrumentation's best knowledge. Preserve gen_ai.request.model when available, then put the returned model in gen_ai.response.model. The upstream destination uses server.address; failures use a low-cardinality error.type.

A proxy creates an attribution wrinkle. The application calls the gateway address, while the gateway calls a provider address. The specification says client-side server attributes are for the server behind intermediaries when available. The gateway span should therefore name the actual upstream endpoint. The incoming HTTP span already identifies the gateway.

AI gateway observability covers the broader signal categories. This implementation guide owns the exact convention boundary.

Policy evaluation deserves its own child span

A provider inference span answers which model operation happened and how long it took. The gateway's policy decision has a different lifecycle. It begins after identity and route context arrive, includes classification and rule evaluation, and ends with allow, modify, or deny. A denied request may produce no provider span at all.

Create a child span such as ai.gateway.policy.evaluate under the incoming gateway request. Use organization-controlled attributes for policy version, decision, route, data class, and rule identifier. Keep values low cardinality where the telemetry backend expects aggregation. Put the user or agent binding in the security decision record and use a pseudonymous subject reference in the trace when operational search requires it.

My opinion is that forcing a deny event into gen_ai.response.finish_reasons is schema theater. The model never generated a response. The policy span should say the gateway denied the request, while the absent provider span tells the rest of the story.

Policy as code for AI supplies the versioned rule artifact that the span references.

Content fields require an explicit privacy decision

The GenAI span specification marks system instructions, input messages, output messages, and tool definitions as opt-in content. That status should survive contact with an AI gateway. Prompt bodies can contain credentials, source code, health records, legal material, and customer data. Copying them into a trace backend creates a second disclosure route and a new retention problem.

Default to metadata. Capture operation and provider, requested and returned model, token counts where supported, duration, error class, route, and policy outcome. Add content only for an approved environment and purpose, under a dedicated data classification and retention policy. Sampling does not erase the sensitivity of the sampled record.

A concrete review screen should show a gray content.capture=false field beside the trace ID and policy version. That small flag gives an incident responder a clear answer before they open the telemetry store. If content capture turns on for a test route, record the approval and expiration time.

The PII redaction guide covers transformation on the actual model request. Telemetry redaction is a separate control applied to copied diagnostic data.

Metrics need low-cardinality dimensions

The official GenAI metrics specification defines client and server instruments with standard attributes. The client operation duration metric uses seconds and explicit bucket guidance. Current conventions also separate inference, embeddings, tool execution, agent invocation, and retrieval into operation-specific instruments where applicable.

At the gateway, use standard provider, operation, model, server, and error dimensions where the specification permits them. Keep identity, request ID, prompt hash, and policy rule detail out of metric labels. Those values create a series for each request or user and can overwhelm a metrics backend. Traces and decision records carry request-level detail.

Gateway-specific metrics can count policy outcomes by bounded dimensions such as route, policy version, decision, and data class. Review the number of active values before release. Model names can also expand unexpectedly when teams allow arbitrary provider strings. Normalize them at configuration time and keep the raw provider response in the trace.

A metric should answer an operational trend question. A trace should reconstruct one execution. A signed decision record should prove the enforcement result independently of the telemetry pipeline.

Development status changes the rollout plan

Many GenAI attributes in the current repository carry development status, while general attributes such as error.type and server.address may be stable. Instrumentation teams should expect names, requirement levels, and operation-specific guidance to change. Pin the semantic convention version used by the gateway and publish it as resource metadata or deployment documentation.

Build a translation layer between internal event fields and exported OTel names. The policy engine can keep a stable internal contract while exporters map that contract to the selected semantic convention release. When a convention changes, update the exporter and run fixtures against both old and new queries during a migration window.

The acceptance suite should cover an allowed nonstreaming request, an allowed streamed request, an automatic retry, a provider error, and a gateway denial. Confirm that a denial has a policy span and no upstream model span. Confirm that retries remain inside one logical provider operation. Verify the response model when a provider aliases the requested model.

The suite checks trace shape against actual request behavior.

Audit evidence stays independent of OTLP export

OTLP export can fail, sample records, transform attributes, or apply short retention. Those are acceptable engineering choices for observability. A regulatory or security decision record needs a separate write path with integrity protection and a retention rule aligned to its purpose.

Generate one request identifier at the gateway boundary. Put it on the policy span, upstream model span, response header where appropriate, and signed decision record. Preserve the OTel trace ID as a correlation field rather than treating it as the sole audit key. Trace IDs can come from an upstream caller, so the enforcement point should validate format and bind the value to its own request record.

AI audit log immutability covers storage and integrity. The OTel exporter remains nonblocking for engineering telemetry. The audit decision follows the gateway's fail-closed contract where policy requires a durable record before forwarding.

DeepInspect

DeepInspect sits on authenticated HTTP traffic deliberately routed to an LLM. It can create a gateway policy span, propagate trace context into the provider call, and map model operations to the current OpenTelemetry GenAI conventions. Content capture remains opt-in, while identity-aware policy and classification stay attached to the enforcement decision.

Each routed request also produces a signed, tamper-evident decision record on an independent path. The shared request identifier lets engineers move between the OTel trace and the security evidence without treating sampled telemetry as the system of record. Book a technical deep dive at deepinspect.ai.