What overhead does an AI gateway actually add?

A well-built gateway adds single-digit to low-tens of milliseconds of forwarding overhead. LiteLLM reports about 7.5 ms for its Python proxy and targets 0.05 ms for its Rust rewrite, Bifrost claims microsecond-level overhead in a mock stress test, and Portkey's real-world figure with features enabled is closer to 20 to 40 ms. All of these are small against an LLM inference baseline of 500 ms to 5 seconds, so gateway overhead is typically a low-single-digit percentage of total request time.

Why are vendor benchmarks misleading?

Because most of them measure proxy forwarding against a mock upstream, which removes the provider's response time, the largest real-world variable. A headline like 40 times faster is measuring microseconds of proxy work on top of a mock, not the request a user experiences. Vendor benchmarks also tend to quote p50 rather than p99, and to test without streaming, both of which flatter the number. Read every benchmark with its upstream, its percentile, and its streaming mode attached.

Which metric should I optimize for?

TTFT under streaming for user-facing workloads, because it is what the user feels, and gateway-added overhead measured as gateway latency minus direct-to-provider latency at matched concurrency for evaluating the gateway itself. Optimizing raw proxy throughput against a mock rarely changes the real request, because provider TTFT dominates. The metric that matters for a security gateway is whether policy evaluation and audit commit stay off the critical path.

Does inline policy enforcement hurt latency?

Not when it is designed correctly. Purpose-built classifiers run in tens of milliseconds, and running identity, policy, and classification in parallel means the added latency is the slowest single check rather than their sum. Caching decisions per identity and route, and committing the audit record asynchronously, keep enforcement off the hot path. The design mistake that does hurt latency is using a full model as a judge synchronously, which adds seconds per check.

How does gateway overhead compare to provider variance?

Provider variance usually exceeds gateway overhead. Second-to-second P99 swings at major providers can move by hundreds of milliseconds, while a well-built gateway adds around 1 to a few tens of milliseconds. This is why shaving proxy overhead below a millisecond has little effect on real request latency, and why the honest way to present a gateway's cost is as a fraction of a provider call whose own timing fluctuates by more than the gateway's entire budget.

AI Gateway Latency Benchmarks: Reading the 2026 Numbers Without Getting Fooled by the Mock Upstream

The published AI gateway latency numbers for 2026 span five orders of magnitude, from Bifrost's claimed 11 microseconds of overhead to LiteLLM's older figure of 40 milliseconds of median overhead with callbacks enabled. That range alone should make an engineer suspicious, because it is not measuring the same thing. Almost every one of these benchmarks clocks proxy forwarding against a mock upstream, which removes the one variable that dominates real requests: the model provider's own response time.

I want to read the 2026 numbers the way a principal engineer should, with the methodology attached to each figure, and then make the point the benchmarks obscure. A well-built gateway adds single-digit to low-tens of milliseconds. A model's time to first token runs from hundreds of milliseconds to several seconds. The gateway is a rounding error on the provider, and the interesting engineering question is not shaving microseconds off the proxy but keeping policy evaluation off the critical path.

The metrics that actually matter

Benchmarks get compared incorrectly because the metric is left implicit. Name it every time.

TTFT, time to first token, from request send to the first token back. Dominated by prompt processing and provider queue. This is what a user experiences as responsiveness.
ITL or TPOT, inter-token latency or time per output token, the steady-state gap between tokens after the first.
Total request latency, which is TTFT plus output tokens times ITL.
p50, p95, p99, the percentiles. p99 is where gateway and provider tail pain shows.
Throughput, sustained requests per second at a stated concurrency.
Gateway-added overhead, the only number this article cares about, defined as gateway end-to-end latency minus direct-to-provider latency at matched concurrency.

If a benchmark does not tell you which of these it measured and against what upstream, the number is not comparable to anything.

The 2026 numbers, with their caveats attached

Here is what the published benchmarks actually claim, and what each one measured.

LiteLLM's June 2026 Rust launch post reports the Python proxy adding roughly 7.5 ms of overhead per request, with the Rust rewrite targeting about 0.05 ms, and throughput rising from 453 to 6,782 requests per second. The test rig is a mock upstream with a microsecond-timing client at ten concurrent clients, and the rig is checked in, which is the right way to publish a benchmark.

Bifrost, from Maxim, claims 11 microseconds of gateway overhead at 5,000 requests per second on a t3.xlarge, and 0.99 ms in a 500-RPS test where it reports LiteLLM at 40 ms. The caveat is load-bearing: the 11-microsecond figure is a Bifrost-only stress test against roughly 10 KB payloads with a mock upstream, and it explicitly excludes upstream response time. The 0.99 ms versus 40 ms comparison uses a 60 ms mock OpenAI response. It is a real proxy-forwarding number and it is not an end-to-end number.

Kong's July 2025 benchmark ran Kong 3.10, Portkey OSS, and LiteLLM against a WireMock upstream at 400 virtual users with 1,000-token prompts, reporting a baseline of 29,005 RPS at P95 24.07 ms and P99 30.35 ms, and stating results only as relative percentages, Kong faster than the others by stated multiples. Mock upstream, proxy-only, no absolute per-gateway overhead in milliseconds.

Portkey markets sub-1 ms latency for its bare proxy, and community reports put real-world overhead with guardrails and routing enabled closer to 20 to 40 ms. Cloudflare AI Gateway, an edge proxy, shows community figures around 10 to 50 ms added. Envoy AI Gateway has no published data-plane overhead benchmark at all; its only 2026 benchmark measures control-plane scaling, route readiness across 2,000 routes, not request latency. Do not cite a request-overhead number for Envoy AI Gateway, because there is not a published one.

The methodology traps

Five traps turn these numbers into misleading comparisons.

Streaming versus non-streaming. A non-streaming test folds TTFT into total latency, and a gateway that buffers the response instead of passing server-sent events through adds latency a non-streaming test cannot see. Measure TTFT under streaming.

Mock upstream versus real provider. LiteLLM, Bifrost, and Kong all benchmark against mocks. That is legitimate for isolating gateway cost, and it deletes the single largest real-world variable, which makes a headline like 40 times faster look decisive when the delta is microseconds sitting on top of a multi-hundred-millisecond provider call.

Cold start. First-request compilation and connection-pool warmup inflate p99, especially on serverless or edge gateways. Report warm and cold separately.

p50 not p99. Vendors quote the median. The tail is where retries, connection churn, and provider variance appear, and where an inline gateway either helps through failover or hurts through an extra hop.

Provider variance dwarfs gateway overhead. Second-to-second P99 swings at OpenAI or Anthropic exceed a well-built gateway's entire overhead budget. A gateway adding around 1 ms to a call whose provider TTFT varies by hundreds of milliseconds is not the latency story, and LiteLLM's own post concedes that gateway overhead is usually a small fraction of total model latency.

Put the numbers in proportion

Frame every gateway figure against the baseline. LLM inference takes 500 ms to 5 seconds. Fast, small models post TTFT under roughly 600 ms on public leaderboards like Artificial Analysis, and frontier reasoning configurations run far higher because of extended reasoning. Against that, a gateway overhead of single-digit to low-tens of milliseconds is between a fraction of a percent and a few percent of the request. The engineering that matters is not the proxy's forwarding speed. It is whether the gateway's own policy work stays off the critical path, which is the reason inline enforcement is affordable at all.

Where policy evaluation adds latency, and how to keep it off the path

Policy and classification cost is real and manageable if the design is right. Purpose-built classifiers run in the tens of milliseconds; practical inline budgets sit around sub-100 ms for an input rail. The trap is using a full model as a judge, which adds 5 to 11 seconds per check and belongs nowhere near a synchronous request. Three patterns keep evaluation invisible:

Parallel evaluation. Run identity, policy, and content classification concurrently rather than in series, so total added latency is the slowest check, not the sum.
Caching. Cache policy decisions per identity and route, and use semantic caching on repeated prompts to remove the upstream call entirely.
Async audit commit. Emit the signed per-decision record off the hot path with a write-behind commit, so producing the audit record never adds to TTFT.

Enforce inline, commit evidence asynchronously. That sentence is the whole latency design for a security gateway.

DeepInspect

DeepInspect is a stateless proxy at the AI request boundary, and it is built to the proportion the benchmarks obscure. Its end-to-end enforcement overhead measures under 50 ms in internal testing, against an LLM inference baseline of 500 ms to 5 seconds, so the enforcement decision is a small fraction of the request the user is already waiting on.

The design follows the three patterns above. Identity binding, per-route policy, and prompt classification run in parallel rather than in series. Policy decisions cache per identity and route. The signed per-decision audit record commits on a write-behind path, so evidence never sits on the critical path to the model. That keeps the gateway's own cost inside the noise floor of provider variance while still enforcing inline and failing closed. If you are evaluating gateway latency for a regulated deployment and want to see the enforcement overhead against your own traffic, book a demo today.