LLM API Retry-After Policy: Safe Delays Without Duplicate Work
An LLM API Retry-After policy must combine the server's delay signal with method semantics, an idempotency contract, a bounded retry budget, jitter, and a final failure state. This guide defines handling for 503, temporary 413, and other documented responses, then turns the policy into a production acceptance fixture with evidence for every attempt and downstream effect.

A retry policy decides more than delay. It decides which response can trigger another attempt, which request can be repeated, how long the caller may keep trying, and how duplicate work is prevented. For LLM APIs, that last decision covers inference charges, batch creation, file operations, and tool effects initiated around a model call.
I want a retry trace that an incident responder can read left to right: attempt one at 09:14:02, a 503 carrying Retry-After: 12, a scheduled second attempt, and one final outcome tied to the same operation identifier. A loop hidden inside an SDK gives weak evidence.
TL;DR
- Parse Retry-After as either an HTTP date or a non-negative number of seconds, then combine it with the caller's deadline and retry budget.
- Repeat a non-idempotent LLM API operation only under a documented idempotency or reconciliation contract.
- Add jitter, cap the delay and attempt count, and stop when the request deadline expires.
- Record every attempt, received status, Retry-After value, chosen delay, operation identifier and destination, plus the final disposition.
Retry-After is a server signal, not a complete algorithm
RFC 9110 Section 10.2.3 defines Retry-After as either an HTTP date or a number of seconds to delay after receiving the response. With 503, the field indicates how long the service expects to remain unavailable. With a 3xx response, it specifies the minimum wait requested before the redirected request.
The client still needs local policy, including parsing for both forms. Reject malformed values as unusable input and fall back to the route's documented backoff rule. For an HTTP date, calculate delay against a trusted clock and clamp a past date to zero. Account for clock skew in telemetry because a date-based value can appear negative on one host and positive on another.
Set an upper delay cap for every route. A provider can ask for a wait longer than the user's request deadline or a batch worker's lease. The caller should end with a typed final state rather than sleeping past the operation's useful life. Preserve the received value and the chosen delay so an operator can distinguish provider instruction from local policy.
The AI gateway latency guide covers request-time budgets. Retry policy consumes part of that budget and must remain visible as a separate component.
Status handling needs an explicit allowlist
RFC 9110 Section 15.6.4 defines 503 Service Unavailable for temporary overload or scheduled maintenance and permits a Retry-After field. The same specification allows Retry-After on a temporary 413 Content Too Large response. The two statuses carry different semantics. A 503 may succeed later with the same request. A temporary 413 needs a documented reason why capacity changes with time.
Build a status matrix for the route. Each row should include the status, required or optional response fields, retry eligibility, delay source, maximum attempts, and final client error. Include connection failures and timeouts separately because they arrive without an HTTP response.
429 handling requires the API's documented contract. Providers often use it for rate limits, but the retry window and accounting unit can vary by account, model, region, or token quota. Treat provider-specific rate-limit fields as inputs to a route adapter. Keep the generic retry engine tied to normalized fields such as eligibility, earliest retry time, quota scope, and source.
Authentication and authorization responses should stop the loop. A repeated 401 with the same credential is credential churn. A 403 indicates refusal of the understood request. The post-authentication gap guide explains why a valid identity still needs a separate route decision.
My blunt view is that "retry every 5xx" is an outage multiplier masquerading as resilience. A gateway error can hide an unknown upstream outcome. Repetition needs both status eligibility and operation safety.
Idempotency decides whether repetition is safe
RFC 9110 Section 9.2.2 defines an idempotent method by the intended server effect of repeated identical requests. PUT, DELETE, and safe methods are idempotent under HTTP semantics. The specification advises against automatic retries for non-idempotent methods unless the client knows the request semantics are idempotent or can determine that the original request was never applied.
Many LLM operations use the POST method. A chat completion may be read-like from the application's perspective, yet it can consume billable compute and emit a distinct result on each attempt. Batch creation, file upload, fine-tuning job creation, and an agent run can create durable resources. Tool calls around the model response can add external effects.
Define one of three postures for each operation:
- Repeatable: the operation has no durable effect beyond separately recorded usage, and duplicate responses are acceptable.
- Keyed: the server or application binds retries to an operation key and returns the existing operation or result for that key.
- Reconciled: the client queries operation state after an ambiguous outcome before deciding on another creation attempt.
The identifier must represent the logical operation, survive process restarts, and remain stable across retry attempts. Scope it to the correct tenant and route. Store the request digest or selected immutable fields beside it so accidental key reuse with different content produces a conflict instead of a false match.
Backoff must stay inside one bounded budget
A route policy should set maximum attempts, maximum elapsed time, per-attempt timeout, delay cap, and jitter method. Retry-After establishes the earliest acceptable time when the route honors it. Local exponential backoff can choose a later time, subject to the operation deadline. Jitter spreads concurrent clients so they avoid returning in one sharp wave.
The deadline belongs to the logical operation, and each attempt consumes part of it. DNS, connection establishment, TLS, request upload, provider processing, and response streaming all count. Starting a new attempt with too little time for a useful result wastes capacity and complicates cancellation.
Respect caller cancellation throughout the retry sequence. A closed browser tab may cancel an interactive request, while a durable batch worker may continue under a separate job lease. Document that distinction in the route contract. For streaming responses, decide what happens after the first response bytes arrive. A generic full-request replay can duplicate text already delivered to the user. Resume requires a protocol-specific cursor or application contract; absent one, end the stream with a visible partial-result state.
The platform engineer AI risk checklist treats retry and fallback behavior as release evidence. Retry is one response, alongside fail closed, bounded fallback, reconciliation, and operator escalation.
Error bodies should preserve machine semantics
Use a consistent final error after the retry budget ends. RFC 9457 Section 3 defines the application/problem+json model. A route can expose a stable problem type for exhausted upstream retries and include extensions such as attempts, elapsedMs, lastStatus, and retryable.
Keep provider-specific data under a controlled extension or internal event. Raw upstream bodies may contain request fragments, account identifiers, implementation details, or inconsistent schemas. The client needs a stable type and corrective action. Operators need a correlation value that joins the normalized response to the attempt ledger.
If the final response includes Retry-After, define whose instruction it represents. Passing through a stale value from an earlier attempt misleads the caller. Recalculate or omit it according to the route contract. Keep the actual HTTP status aligned with any status member in the problem body.
The acceptance fixture needs ambiguous outcomes
Run the retry policy against a controllable upstream stub through the deployed load balancer. A green test should show timing and effects, not merely a successful final response. Include these cases:
- 503 with delay-seconds, followed by success.
- 503 with an HTTP date, including a controlled clock-skew case.
- A malformed Retry-After value and the documented fallback delay.
- A delay beyond the operation deadline, producing the final typed failure.
- Two simultaneous callers sharing one keyed operation.
- A connection drop after the upstream creates a resource but before the client receives the response.
- A partial stream followed by connection failure.
- A repeated 401 and a 403, both ending without an automatic loop.
- Process restart between attempts, proving that the operation key and budget survive.
Place a visible counter in the upstream stub and retain its event log. One keyed logical operation should create one durable resource even when two client processes retry. The attempt ledger should show each request separately and point to one final disposition.
Also test overload behavior in the retrying tier. Limit concurrent retries and queue depth. Shed work when that tier reaches its own ceiling. Otherwise a provider recovery event can trigger a second overload inside the client fleet.
DeepInspect
DeepInspect can apply policy and produce decision evidence for authenticated user or agent traffic routed through its HTTP path to an LLM endpoint. That record can identify the caller, route, destination, policy version, and allow or deny outcome for each routed attempt. It can also support controls that restrict unapproved provider fallbacks.
The application or API client still owns retry scheduling, operation keys, reconciliation, cancellation, and any external side effects. Provider quota accounting and direct traffic outside the routed path remain with their respective owners. Correlate the client attempt ledger with DeepInspect decision records rather than treating one layer as proof of the other.
Book a demo today.