LLM API Payload Size Policy: Bound Requests Before Inference
An LLM API payload size policy needs separate limits for HTTP content, decoded fields, files, images, messages, tokens, and downstream expansion. This guide defines a deterministic rejection contract, a test matrix for every boundary, and the evidence an engineering team should retain. It also separates gateway controls on routed HTTP traffic from limits owned by model providers, storage services, and local runtimes.

An LLM API request can fit beneath a reverse proxy's byte ceiling and still exceed the model route's operating limit after decoding or file retrieval, then image expansion or tokenization. A useful payload policy measures each stage separately. I want the release artifact to show the exact unit and threshold, plus the rejection owner and response contract for every stage. A single max_body_size value leaves too much hidden.
Picture the acceptance test at 16:40 on release day: two terminal panes show requests that differ by one byte, while the evidence record identifies the gateway rule that accepted one and rejected the other. That concrete boundary is the standard.
TL;DR
- Set independent ceilings for HTTP content, decoded fields, attachments, images, message count, tokens, and downstream expansion.
- Reject oversized content at the earliest authoritative layer and return 413 with a stable machine-readable problem type.
- Test the exact limit and one unit on either side of it. Repeat with compression, chunked transfer, streaming, and file references.
- Record the caller, route, measured value, unit, policy version, outcome, and correlation ID without retaining sensitive prompt content by default.
Payload size has several measurement points
The first number is the HTTP request content received by the edge or gateway. RFC 9110 Section 15.5.14 defines 413 Content Too Large for content that exceeds what the server is willing or able to process. That status gives the HTTP layer a clear refusal signal. It says nothing about the model's token window or an application's attachment policy.
The next measurements belong to the application contract. Count JSON bytes after content decoding and measure both the number and length of messages. Measure the size of each string field. Measure inline base64 data after decoding too. A 6 MB encoded image and its decoded binary have different sizes, so the policy must name which value controls acceptance.
File references create another branch. The initial JSON may contain only a short object ID. Retrieval can pull a much larger document into preprocessing. Record limits for each file and for the combined retrieved content. Image and audio routes need byte limits alongside limits for dimensions, duration, and count. Model token limits come after serialization and tokenization, using the tokenizer tied to the selected model version.
The AI gateway file upload scanning guide covers inspection and quarantine. Size policy should run before expensive scanning where the receiver can safely determine the limit, then run again after decoding or retrieval when the relevant measurement becomes available.
HTTP field limits and content limits need separate rules
Header fields can become large before the request body begins. RFC 9110 Section 5.4 states that HTTP sets no predefined field-length ceiling and requires a server to return an appropriate 4xx response when received fields exceed what it wishes to process. The specification warns that silently ignoring oversized fields increases exposure to request smuggling.
Treat header-section limits as a parser and routing safeguard. Treat content limits as an application resource policy. The evidence should identify which boundary fired. A generic 400 for every size condition forces the client and operator to guess.
For request content, use 413 when the submitted content itself exceeds the receiver's accepted size. A temporary capacity restriction may include Retry-After, as RFC 9110 permits for temporary 413 conditions. Permanent route limits should give the client a corrective value in a typed error body, such as the accepted byte ceiling or maximum attachment count.
Compressed content needs an explicit stance. Set a wire-size limit and a decoded-size limit. Stop decompression once the decoded ceiling is crossed. This avoids allocating the full expanded object before refusal. Also set limits on nesting and array length. Set individual field limits too, because a byte ceiling alone leaves parser work and application fan-out unbounded.
Token limits belong to the selected model route
Token count depends on the model and request assembly. The user-supplied prompt may gain a system instruction, conversation history, retrieved passages, tool definitions, image descriptors, and response-budget reservation before the provider receives it. Measure the final assembled request that will cross the model boundary.
Create a route manifest with these fields:
- HTTP content limit in bytes at the receiving gateway.
- Decoded JSON limit and maximum nesting depth.
- Limits for each field and message, including the message-count limit.
- Per-file and aggregate attachment limits.
- Image limits for count and dimensions, with a separate decoded-byte limit.
- Input-token ceiling tied to the tokenizer for the named model identifier.
- Reserved output tokens and total context ceiling.
- Retrieval and tool-definition expansion ceilings.
The OWASP API Security Top 10 for LLM endpoints places resource consumption at the API layer. The payload policy turns that category into measured route decisions before a provider-specific overflow becomes the only signal.
My blunt view is that silently truncating a prompt is data corruption with polite branding. Truncation changes instructions and evidence. If the product intentionally shortens content, define that transformation as a named operation and expose it to the caller. Record both the original and resulting measurements.
Rejections need one stable machine contract
A size rejection should carry an HTTP status that matches the actual response and a body the client can handle without parsing prose. RFC 9457 Section 3 defines application/problem+json with members for type, title, status, detail, and instance. Extensions can carry structured fields specific to the problem.
Define a stable type URI for each policy category or one documented payload-limit type with extensions. Useful extensions include limit, observed, unit, component, and policyVersion. Keep detail focused on correction. Avoid stack traces and provider credentials. Exclude internal file paths and raw prompt fragments too.
The HTTP status and problem body's status must agree. RFC 9457 treats the body member as advisory, while generic HTTP software follows the response status. A mismatch turns an ordinary client error into an incident-analysis dispute.
For a multi-part request, identify the component that crossed the limit. attachment[2] or a JSON Pointer can locate the failing part without copying its content. The AI audit log schema guide provides a broader event structure. Add size-policy fields to the decision record rather than inventing an isolated log stream.
The acceptance fixture should force every boundary
Build a fixed fixture for each route and run it against the deployed topology. For every numeric ceiling, send requests at the threshold and one unit on either side of it. Record the observed status and problem type, plus the measured value and policy version.
Add cases that expose disagreement between layers:
- A small compressed body that expands beyond the decoded ceiling.
- Chunked transfer whose cumulative content crosses the limit.
- Valid JSON with an oversized string and another with excessive nesting.
- Several permitted files whose aggregate exceeds the request ceiling.
- A short file reference that retrieves an oversized object.
- A prompt under the user limit that exceeds the token ceiling after retrieval.
- An over-limit streaming request where the receiver stops processing promptly.
Run the same tests through the public load balancer and any service mesh. Include the gateway, application server, and model adapter. A test against one process misses the real disagreement risk. Preserve configuration digests for every enforcing layer. The release passes when the documented owner rejects each over-limit case and the application avoids forwarding it downstream.
Operational ownership keeps the policy coherent
Platform engineering owns transport and parser ceilings. The application team owns schema and attachment rules. The AI platform owner maintains model and token limits. Security reviews decompression and parser-abuse controls, while product owners approve any user-visible transformation such as summarization or truncation.
Put the numbers in one versioned route manifest even though enforcement spans several components. Changes to the model alias, tokenizer, retrieval fan-out, accepted media, or reverse proxy reopen the relevant tests. Provider documentation can set an upper bound, but your route may choose a lower one for cost and latency constraints or data-handling reasons.
Measure rejected requests by route and policy version. Alert on abrupt changes in over-limit events and repeated attempts just above a threshold. Also alert on disagreement between gateway acceptance and provider rejection. Keep request content out of operational metrics. Hashes, sizes, identities, decisions, and correlation values usually provide the evidence needed for diagnosis.
DeepInspect
DeepInspect can enforce payload policy on authenticated user or agent traffic deliberately routed through its HTTP path to an LLM endpoint. At that boundary, it can evaluate supplied identity context and destination, plus content classification and configured request limits. It can then produce a per-decision audit record for the allow or deny outcome, including any redaction.
Provider-side upload stores and local model processes retain separate owners. STDIO tools and direct calls that bypass the routed HTTP path do too. DeepInspect also depends on the application to supply trustworthy identity context and to route the relevant request through the enforcement point. Use the route acceptance fixture to prove those conditions in the deployed topology.
Book a demo today.