Gateway API Inference Extension: Production Routing Guide

Gateway API Inference Extension changes the backend decision inside a Kubernetes inference request. A normal Gateway API route can send traffic to a Service and let the data plane spread requests across ready endpoints. The extension introduces InferencePool, which groups model-serving Pods and can consult an Endpoint Picker before choosing a replica.

That extra decision matters because two healthy Pods can carry very different inference costs. One may hold the requested LoRA adapter in memory. Another may have a long decode queue or a cold prefix cache. I want to walk through the resource model, failure policy, request path, and production tests that make this routing layer predictable.

TL;DR

• InferencePool is a stable inference.networking.k8s.io/v1 backend for Gateway API routes.
• An Endpoint Picker can select a model-serving Pod using inference metrics instead of ordinary connection balancing.
• HTTPRoute still chooses the pool; the picker chooses a compatible endpoint inside that pool.
• failureMode needs an explicit availability decision and a tested response when the picker disappears.
• Scheduling belongs to the inference gateway. Identity-bound prompt policy and audit evidence require a separate HTTP control.

The extension adds an inference-aware backend

The official Gateway API Inference Extension overview describes an inference gateway as a proxy or load balancer coupled to endpoint selection. Gateway API still supplies the familiar listener and route model. The extension adds model-serving resources plus a protocol between the data plane and the routing component.

InferencePool is the main backend object. It selects a group of Pods that share the relevant serving shape, such as the accelerator type and base model. The pool also names target ports. When configured, endpointPickerRef points to the service that makes the per-request endpoint decision.

An infrastructure team can own Gateway and HTTPRoute, while the inference platform team owns pool membership and picker behavior. The AI gateway architecture guide covers the larger HTTP proxy design around that routing layer.

InferencePool is stable, with version details that matter

The current InferencePool documentation marks the resource generally available since v1.0.0. Its API version is inference.networking.k8s.io/v1. The selector must match the labels on the model-server Pods, and targetPorts tells the gateway where those Pods accept inference traffic.

One version detail belongs in every platform runbook. Before Gateway API Inference Extension v1.5.0, endpointPickerRef was required. It is now optional, which permits an implementation to support InferencePool without a user-managed picker. Support varies by inference gateway, so omission should be an implementation decision backed by its documentation.

My preference is explicit configuration in production. A missing picker reference should be intentional and visible in review. Silent fallback from inference-aware selection to ordinary endpoint balancing creates a capacity incident that can hide behind green Kubernetes readiness checks.

HTTPRoute chooses the pool before endpoint selection begins

The route still performs the first routing decision. Its backendRefs entry uses group inference.networking.k8s.io, kind InferencePool, and the pool name. A path match on / can therefore send OpenAI-compatible requests to a pool rather than a Service.

A request crosses these decisions in order:

1. Gateway processing matches the listener and HTTPRoute rule.
2. That route selects an InferencePool backend.
3. With a picker configured, the data plane sends it request information for the selected pool.
4. An eligible model-server endpoint comes back as the chosen destination.
5. Finally, the gateway forwards the HTTP request to the selected Pod.

The implementer's guide defines x-gateway-destination-endpoint as the selected destination signal. It also supports a fallback endpoint in metadata. A gateway implementation must keep the header and structured metadata values aligned.

Endpoint Picker quality determines routing quality

An Endpoint Picker can inspect serving information that a generic load balancer lacks. The project documentation names KV-cache utilization and pending queue length. It also calls out active LoRA adapters. Those signals let the picker avoid a replica that is technically ready but expensive for the current request.

The repository includes a Lightweight Endpoint Picker for conformance testing. The implementer's guide explicitly advises against treating that component as a production scheduler. A production deployment needs a supported picker with known behavior under streaming traffic, stale telemetry, rapid Pod turnover, and picker restarts.

A six-Pod dashboard can stay green while one GPU accumulates a queue and another holds the adapter every request needs. That is the operational argument for the extension in one picture. The routing decision should follow serving state, rather than stop at Kubernetes readiness.

FailureMode is an availability policy

The documented InferencePool example sets endpointPickerRef.failureMode to FailOpen. If the picker fails to select an endpoint or stops responding, the gateway can forward the request to an endpoint selected by its own fallback behavior. That preserves availability while sacrificing the picker's optimization for the affected request.

Treat this field as an incident policy. FailOpen can make sense when any ready replica can serve the same model and the cost of a slower response is acceptable. A stricter posture may be appropriate when adapter presence or another serving capability is required for correctness. Confirm the values supported by the chosen gateway implementation, then test the actual data plane response.

The test is simple: remove network access to the picker while requests are active. Record the status code and destination behavior. A configuration review alone cannot prove the runtime path.

Production acceptance tests need observable outcomes

Start with a request carrying a known model name and a trace identifier. Confirm that HTTPRoute selects the intended pool, the picker observes the request, and the final Pod receives the same trace context. Then change one serving condition at a time.

Use these acceptance checks:

• Mark one endpoint unavailable and verify that the picker excludes it before the next request reaches that Pod.
• Create queue pressure on one replica, then confirm new work moves to a less-loaded eligible endpoint.
• Load a LoRA adapter on a single Pod and send a request naming that adapter.
• Restart the picker during a streamed response and observe the configured failure behavior.
• Return an unknown endpoint from a test picker and verify rejection rather than arbitrary forwarding.
• Rotate Pods quickly enough to expose stale endpoint data, then measure how long the picker takes to converge.

Capture gateway access logs beside picker decision logs. The pair should explain which pool matched and why one endpoint won.

Security policy sits beside scheduling

Gateway API Inference Extension optimizes routing for self-hosted model traffic. Its picker decides where a request should run. That decision does not establish that the caller may send the prompt, that the selected model may receive the data class, or that the response may return to the user.

Place identity-aware policy on the authenticated HTTP path before the inference request reaches the pool. The policy layer should receive caller context from the application, inspect prompt and response content, and bind the result to the route plus policy version. The identity-aware gateway architecture explains that request-level authorization boundary.

Keep the ownership split visible. Kubernetes RBAC governs who can modify the resources. NetworkPolicy constrains Pod communication inside the cluster. Image provenance and node security protect the serving infrastructure. Model artifact integrity belongs to the model supply chain. The inference gateway schedules requests; each adjacent control retains its own owner.

DeepInspect

DeepInspect can sit before a Kubernetes inference gateway on the authenticated HTTP request path. It evaluates application-supplied identity and role, inspects prompt content, applies per-route policy, and records the permit, redact, or deny decision before the request reaches an InferencePool. Response inspection runs on the return path.

The Endpoint Picker still owns replica selection. DeepInspect does not schedule GPUs or repair a failed picker. This separation keeps serving optimization independent from the policy record and lets teams combine Gateway API Inference Extension with the same enforcement model used for hosted LLM routes. The Envoy AI Gateway comparison shows how those layers compose in one Kubernetes data path.

Book a technical deep dive at deepinspect.ai.