← Blog

Safe Idempotent LLM Retries Need an Operation Ledger

Parminder Singh
Parminder Singh··7 min read
Summarize with AI

Safe idempotent LLM retries require a logical operation identifier, a stable request fingerprint, a bounded attempt budget, and reconciliation after ambiguous outcomes. This guide separates transport attempts from business effects, applies HTTP method semantics to OpenAI POST requests, and defines the ledger and release fixtures needed to prevent duplicate jobs, charges, and downstream actions.

Platform & Architectureai-securityllm-securityarchitecturedevsecopsauditpolicy-enforcement
Safe Idempotent LLM Retries Need an Operation Ledger

An LLM client loses the connection after sending a POST. The upstream may have accepted the request and generated output before the client saw a response, with usage already charged. A second POST can create another effect. Safe idempotent LLM retries start by treating that silence as an ambiguous outcome, then resolving it through a logical operation record rather than optimistic repetition.

I want each retry sequence to leave one readable ledger row: operation op_7f2, request fingerprint 9b4..., three transport attempts, one accepted upstream result, and one business effect. That record is the center of the design.

TL;DR

  • Give the logical operation a stable ID and bind it to a canonical request fingerprint before attempt one.
  • Retry only under an explicit operation contract. The operation must be repeatable or provider-keyed, with reconciliation required after an ambiguous outcome.
  • Honor OpenAI Retry-After guidance, add jitter, and keep every attempt inside one deadline and attempt budget.
  • Commit downstream effects once, then retain provider request IDs and final disposition in the operation ledger.

HTTP idempotency describes the intended server effect

RFC 9110 Section 9.2.2 defines an idempotent method by the intended server effect of multiple identical requests. PUT and DELETE carry that property under HTTP semantics, as do safe methods. The RFC permits automatic repetition after a communication failure because the intended effect stays the same even when responses differ.

The same section places a stricter boundary around a non-idempotent method. Automatic repetition needs knowledge that the operation semantics are idempotent, or a way to determine that the original request was never applied. A proxy also has a direct prohibition against automatically retrying non-idempotent requests.

OpenAI model operations commonly use POST. Method alone therefore provides zero permission to replay. The application needs an operation-level contract covering the specific route and SDK behavior, plus provider behavior and the downstream action. A response-generation call may consume billable compute twice. Batch creation can produce two durable jobs. An agent loop can execute a tool after each generated result.

The LLM API Retry-After policy covers delay parsing and status eligibility. Idempotency supplies the separate answer about duplicate effects.

One logical operation can contain several attempts

Create the operation record before the first provider call. Its identifier should survive worker restarts and remain stable across every attempt. Scope it to the tenant and business action, with the route included so two customers can use the same local order number safely.

Bind that identifier to a request fingerprint. Canonicalize only fields the contract declares immutable, then hash the canonical representation. Include the provider route, model, tool configuration, and selected generation parameters when they affect the operation. Store the fingerprint beside the operation ID. A later attempt presenting different content under the same ID should receive a conflict state.

Keep attempt data in child records. Each attempt needs a start time, deadline, destination, HTTP status or transport error, provider request identifier, received Retry-After, chosen delay, and response fingerprint where available. The parent record owns the final disposition and business-effect state.

My blunt opinion is that a retry counter inside an SDK is bookkeeping, not an idempotency design. The counter disappears at the exact moment a crashed process makes the outcome hardest to reconstruct.

The AI audit log schema offers a useful model for stable correlation fields. Keep the retry ledger operationally separate while preserving a join key.

Useful retry cases need an explicit contract

A route should choose one posture and name it in the client adapter.

A repeatable operation permits duplicate provider executions, and downstream code selects one result. This can fit disposable evaluation work where cost and nondeterministic output are understood.

A provider-keyed operation uses a provider-documented key that binds repeated submissions to one operation or resource. Preserve the key across attempts and test its scope, retention period, conflict behavior, and response replay semantics.

A reconciled operation moves an ambiguous submission into an inquiry state. Before another creation attempt, the client checks the application ledger and either a provider resource or callback record.

Reconciliation needs a terminal decision. The worker may attach an accepted resource to the operation. It may instead prove that submission failed before acceptance. If the inquiry budget expires first, the operation ends in manual_review. Blindly converting an unknown state into another POST defeats the ledger.

For OpenAI calls, retain the provider request ID whenever a response reaches the client. The OpenAI error guide distinguishes rate, spend, quota, server, and overload failures. Spend and quota errors require a limit change rather than repetition. The guide directs clients to follow Retry-After for applicable 429 and 503 responses.

Backoff belongs to the operation budget

OpenAI's rate-limit guide recommends random exponential backoff for rate-limit recovery and points out that unsuccessful requests consume per-minute capacity. This makes unbounded retry traffic self-defeating.

Set a maximum elapsed time and attempt count. Add a per-attempt timeout and delay cap. The logical deadline begins before attempt one. DNS, TLS, upload time, provider processing, response streaming, reconciliation queries, and sleeps all consume it. Honor a valid Retry-After as the earliest retry time, then apply local backoff and jitter inside the remaining budget.

Interactive and durable work need different budgets. A browser request may end when the user disconnects. A batch worker can continue under a durable lease, provided the operation owner explicitly allows that behavior. Streaming adds a harder boundary because response bytes may already sit in the user's transcript. A generic replay can append a second answer. Mark the first stream partial and require a protocol-specific resume contract before continuing.

The AI gateway latency guide explains the request-time budget surrounding model calls. Retry time should appear as a named share rather than an invisible extension.

Downstream effects need their own commit rule

Provider idempotency covers the provider operation described by its contract. Application side effects still need a local once-only rule. Email delivery, ticket creation, database updates, payments, and tool invocations can each duplicate even when the model request resolves cleanly.

Use the logical operation ID as the deduplication key at the effect boundary. A transaction can write the accepted result and an outbox entry together. A worker then claims that outbox item and records the effect's external identifier. Consumers should treat repeated delivery of the same item as the same action.

Picture the release review with three index cards on the table. Write provider accepted in blue and mark the white one client timed out. Put tool executed on the red card. Shuffle their arrival order. The system should still produce one final operation state and one tool effect.

Avoid placing prompt bodies or bearer tokens in the ledger. Provider credentials also stay out. Store approved fingerprints and bounded metadata. Content access remains governed by the application's data-handling policy.

Release tests must force ambiguous outcomes

A useful acceptance fixture needs a controllable upstream stub plus a visible effect counter. Run the real client path through the queue and load balancer. Exercise these cases:

  • Drop the connection after the stub accepts the request but before any response reaches the client.
  • Return 429 with Retry-After, then succeed while proving the chosen delay stayed inside the deadline.
  • Return a spend-limit error and prove the operation enters a terminal blocked state.
  • Crash the worker after provider acceptance and before the local final-state update.
  • Start two workers for the same operation ID and request fingerprint.
  • Reuse the operation ID with changed model parameters and require a conflict.
  • Deliver one accepted result twice to the outbox consumer.
  • Stream a partial response, close the connection, and require an explicit partial terminal state.

Count provider acceptances and external effects separately. One may exceed one for a declared repeatable operation. The business-effect counter should match the route's commit contract. Preserve timestamps and IDs so the AI audit log chain-of-custody guide can be applied to retained evidence.

DeepInspect

DeepInspect evaluates authenticated user or agent HTTP requests deliberately routed through its proxy to an LLM endpoint. For each routed attempt, it can apply identity-aware policy to the supplied context and record the caller, route, destination, policy version, classification, and allow or deny result. That provides an independent policy record for the traffic it sees.

The client and application remain owners of operation IDs, request fingerprints, retry scheduling, reconciliation, provider idempotency keys, streaming recovery, and downstream effect commits. OpenAI owns its documented API behavior. Webhook receivers own callback deduplication. Direct provider calls and local tool execution sit outside the routed HTTP boundary.

Join the DeepInspect decision ID to the client attempt record. The combined evidence can show that three attempts belonged to one logical operation while preserving the distinct owner for each effect.

Book a demo today.