← Blog

Envoy ext_proc for LLM Policy: A Production Request-Path Guide

Parminder Singh
Parminder Singh··6 min read
Summarize with AI

Envoy ext_proc can place an external policy service inside an LLM HTTP request and response path. The design depends on processing mode, body buffering, gRPC failure behavior, mutation limits, and a clean identity handoff. This guide maps those choices into a deployment pattern that can deny a model request before upstream transmission and preserve decision evidence without turning the processor into a routing engine.

Platform & Architectureai-securityllm-securityarchitecturepolicy-enforcementinline-enforcementaudit
Envoy ext_proc for LLM Policy: A Production Request-Path Guide

Envoy's External Processing filter opens a bidirectional gRPC stream between the proxy and an external processor. The processor can receive selected HTTP headers and body chunks plus trailers, then return mutations or an immediate local response. That makes ext_proc a useful enforcement hook for LLM traffic, provided the filter sees the complete policy input before Envoy forwards the request upstream.

I would keep the first deployment deliberately boring: one LLM route, buffered request bodies, streamed responses, and a fail-closed processor. A whiteboard full of mode overrides is usually a sign that the policy boundary has become a second router.

TL;DR

  • Use buffered request-body processing when policy depends on the complete LLM JSON payload.
  • Pass verified identity through trusted metadata or headers, then bind it to the model route plus the content decision.
  • Return an immediate response for denied requests before Envoy opens the upstream model path.
  • Stream model responses when inspection can operate incrementally; preserve a final decision record on close or abort.
  • Treat processor timeout and gRPC failure behavior as security policy, with explicit fail-closed tests.

The ext_proc exchange sits inside the HTTP lifecycle

Envoy's External Processing filter documentation defines an external processor as a gRPC service that can examine and modify request or response headers along with bodies and trailers. Envoy sends ProcessingRequest messages over a bidirectional stream. The service answers with matching ProcessingResponse messages.

That exchange gives the processor two enforcement actions. It can continue the filter chain after applying permitted mutations, or return an ImmediateResponse that generates a local HTTP response and stops further filter processing. The second action is the deny path for an LLM policy service.

The physical sequence is visible in a packet capture. The application opens one HTTP connection to Envoy. Envoy opens a separate gRPC stream to the processor. Only after the processor permits the request should Envoy send the model payload to the upstream provider. A denied prompt should produce a local response while the upstream connection remains quiet.

This is a narrower implementation surface than the routing layer covered in DeepInspect versus Envoy AI Gateway. Envoy AI Gateway selects and translates model traffic. ext_proc supplies a hook for an external decision service on a selected HTTP route.

Processing mode decides what the policy service can know

Envoy sends only the message parts enabled by the processing mode. The Envoy Gateway ext_proc task makes this explicit: requests and responses are absent from the external processor by default, and processingMode selects headers and body handling.

For LLM policy, the request side usually needs headers and a complete body. Headers carry the route, media type, correlation identifiers, and trusted identity material. The JSON body carries the model name, messages, tool declarations, attached-file references, and generation parameters. A processor that sees headers alone can authorize a route while remaining blind to the prompt.

Buffered request-body processing fits policies that classify the assembled prompt or validate the complete tool catalog. It delays upstream transmission until the body is available for one decision. Streamed request processing fits large payloads only when the policy can decide incrementally and can safely handle a deny after earlier chunks have arrived. Sending any chunk upstream before the final decision weakens the stop line.

Response handling has a different shape. Server-Sent Events can be inspected chunk by chunk when the response rule is incremental. A rule that depends on the complete answer needs buffering, which removes the latency benefit users expect from streaming. Pick the mode from the rule rather than copying one mode across both directions.

Identity must enter through a trusted handoff

ext_proc receives headers plus dynamic metadata and configured request attributes. Those fields become policy inputs only after another component establishes their trust. An application-supplied x-user-role header is a claim. A JWT validated by Envoy, with selected claims placed into metadata for the processor, is evidence the policy service can evaluate.

The handoff should contain a stable principal identifier and tenant, followed by the role or delegated scope, route name, plus a request correlation identifier. Keep provider credentials out of this metadata. The processor needs enough context to decide if this principal may send this content to this model, while Envoy or the upstream-auth layer retains custody of the provider secret.

The AI gateway architecture guide describes the wider path. At the ext_proc boundary, the practical requirement is simpler: every permit or deny result must name the verified identity input and the policy version that produced it.

Test the trust boundary with a forged header. Send a valid application token for a low-privilege role while adding a high-privilege role header directly. The processor should receive the role derived from the validated token, and the forged value should be stripped or ignored.

Response types should stay narrowly constrained

The ext_proc protocol reference permits header and body mutation, processing-mode and timeout overrides, plus immediate responses. That is considerable authority inside one filter.

An LLM policy processor rarely needs all of it. Give it permission to add a compact decision identifier, redact or replace a body only on routes that require modification, and emit a local deny response. Block mutations to routing and authentication headers unless the design explicitly depends on them. A processor that can rewrite the upstream host or provider credential has crossed into traffic control.

Body mutation also depends on mode. Envoy documents that header mutations returned during body processing take effect only with buffered body processing. The protocol can replace a body chunk, and CONTINUE_AND_REPLACE can replace the remaining message. These features deserve negative tests because an accidental empty mutation can erase the prompt while still returning a successful policy decision.

I prefer a decision service that returns the smallest possible instruction set. Permit or deny, with redaction where the route permits it, is enough for a security boundary. Provider selection and retry plus fallback belong to the gateway layer behind it.

Failure behavior is part of the control

Every request now depends on a gRPC exchange. The deployment must define what happens when the processor is unreachable, returns the wrong response type, exceeds its message timeout, or closes the stream halfway through a model response.

A regulated route should fail closed before upstream transmission. Return a stable application error, record the unavailable policy dependency, and keep the provider request unsent. Lower-risk routes may choose another posture, but the exception should be attached to a named route and owner rather than inherited from a global proxy default.

Run four fault tests before launch:

  • Stop every processor endpoint and send a permitted-looking request.
  • Delay the processor past the configured message timeout.
  • Return a body response when Envoy expects a header response.
  • Close the processor stream after response streaming has begun.

Envoy exposes counters for failed streams and message timeouts, rejected header mutations, plus errors ignored by configured failure behavior. Alert on those counters beside the policy service's permit and deny totals. A green proxy dashboard can hide a processor that has been bypassed by a permissive failure setting.

A production acceptance test needs route evidence

Start with one authenticated POST route to a test model endpoint. The request set should include a permitted prompt, a denied data class, a forged identity header, malformed JSON, an oversized body, and a streamed response that ends early.

For each case, preserve the inbound request identifier and validated principal, followed by the ext_proc decision identifier, active policy version, upstream connection result, HTTP status, plus final stream state. The denied case should show no upstream request. Put the proxy access log beside the processor record on one screen. A reviewer should be able to follow the same identifier without guessing which timestamp belongs to which call.

This acceptance test is distinct from the general AI gateway versus API gateway decision. It proves that one concrete Envoy filter chain enforces the intended LLM route rather than merely loading an extension policy.

DeepInspect

DeepInspect can serve as the identity-aware policy layer for authenticated HTTP LLM traffic deliberately routed through the enforcement path. In an Envoy deployment, ext_proc can hand the verified identity, selected route, model destination, and complete model-bound request to that decision point before upstream transmission.

DeepInspect governs the HTTP request and response that traverse the proxy. Envoy configuration, JWT validation, provider routing, local tools, and direct bypass traffic remain separate controls. For routed requests, the decision record can retain the principal, policy version, data classification, destination, outcome, and correlation identifier used in the Envoy access log.

Book a technical deep dive at deepinspect.ai.

Frequently asked questions

Should LLM request bodies be buffered or streamed to ext_proc?

Buffer the request when the policy depends on the complete JSON payload, including all messages, tools, and file references. Streaming fits policies that can make safe incremental decisions. If Envoy sends content upstream before the processor reaches a final decision, a later deny cannot recover bytes already transmitted.

Can ext_proc inspect streaming model responses?

Yes. Envoy can send response body chunks to the processor. The processor must maintain enough per-stream state for the selected rule and answer according to the configured body mode. Full-response rules require buffering, while token or pattern checks can operate incrementally.

Does ext_proc authenticate the user?

The filter carries inputs to the external processor. Authentication should happen before the processor through JWT validation, mTLS identity, or another trusted mechanism. The processor then evaluates verified claims and route metadata. Raw caller headers should never become identity merely because they reached gRPC metadata.

Where should provider routing run?

Keep provider selection, translation, quota, and fallback in Envoy AI Gateway or another routing layer. The external processor should decide if the authenticated request may proceed under the active policy. Separating those responsibilities makes a denied request and a failed provider route distinguishable in logs.