← Blog

Wasm Plugins in an AI Gateway: Keep Policy Small, Signed, and Observable

Parminder Singh
Parminder Singh··7 min read
Summarize with AI

A Wasm plugin can place request parsing, identity handoff, local denial, and telemetry inside an AI gateway without compiling those decisions into the proxy binary. The useful design keeps provider routing outside the plugin, pins the module artifact, limits host capabilities, pauses model traffic until policy finishes, and records the exact module version behind every outcome.

Platform & Architectureai-securityllm-securityarchitecturepolicy-enforcementinline-enforcementaudit
Wasm Plugins in an AI Gateway: Keep Policy Small, Signed, and Observable

A Wasm plugin runs inside the gateway's proxy host and receives callbacks as an HTTP request and response move through the filter chain. For LLM traffic, that position can expose the authenticated caller and model route to one policy component. The same component can receive the JSON body, streamed output, and final status.

The placement is powerful enough to become dangerous. A plugin that can read and mutate prompts and headers can also make outbound calls and stop traffic. I would reject a deployment where Friday's module update can change model access without a pinned digest and a release record.

TL;DR

  • Put the Wasm filter after caller authentication and before the upstream model route.
  • Pause request processing until any full-body policy decision finishes, then return a local denial or continue upstream.
  • Give the module only the host functions and metadata required by its policy job. Restrict outbound destinations to the same job.
  • Pin the module artifact, canary it in log-only mode, and record its digest with every permit or deny result.
  • Test body limits, streamed responses, plugin crashes, timeouts, and direct bypass routes before production.

The plugin is an HTTP filter with a narrow job

Envoy's official Wasm HTTP filter documentation describes a filter implemented by a Wasm plugin and shows the module loaded through envoy.filters.http.wasm. The current Envoy page also labels the filter experimental and warns that its configuration structures may change. Pin the Envoy release in the acceptance rig and rerun module-loading plus filter-behavior tests before an upgrade.

In an AI gateway, the filter's job should be explicit. It can collect trusted identity metadata, parse a model request, call or run a policy decision, return a local denial, and emit decision telemetry. Provider selection and credential custody should remain with the gateway components that already own them, along with retry policy and fallback routing.

The DeepInspect and Envoy AI Gateway comparison draws that line at the product level. A Wasm module that gradually becomes the router, credential broker, classifier, and audit database has turned one extension into an undocumented control plane.

Filter order establishes the enforcement boundary

Put authentication before the Wasm policy filter. The plugin should receive a principal derived from a validated JWT or mTLS workload identity. Another trusted mechanism can establish the principal before the plugin runs. Raw client headers remain caller claims until that validation happens.

Place the policy filter before the upstream router. The Proxy-Wasm ABI defines request-header and request-body callbacks that can return PAUSE, plus a host function that sends a local HTTP response before the upstream request proceeds. This sequence creates a testable stop line: a denied LLM request reaches the plugin and produces a local response while the provider sees no request.

A diagram can hide the filter order inside one blue gateway box, while the packet trace exposes it. During acceptance testing, put the proxy access log and mock-provider log on the same screen. Send one permitted prompt and one denied prompt, then confirm that only the permitted request opens the upstream path.

The wider AI gateway architecture guide covers identity and policy as separate components, along with routing and evidence. Wasm supplies one implementation hook inside that architecture.

Body processing needs an explicit pause point

The Proxy-Wasm ABI specification defines callbacks for request headers and request-body chunks. It also defines callbacks for response headers and response-body chunks. A plugin can read or modify each available body buffer. When request processing is paused, the host can buffer later chunks instead of forwarding them upstream.

That behavior matters for LLM JSON. A policy that depends on the complete messages array and tool list should pause until the full request body arrives. The same requirement applies to attached-file references. Continuing after the first chunk can release part of the prompt before the plugin reaches a deny decision. The resulting local response arrives too late to recover bytes already sent to the model provider.

Set a body limit before buffering. Reject oversized payloads with a stable status and decision code rather than letting the proxy worker accumulate an unbounded prompt. Record the received byte count and configured limit. Include the route, module digest, and outcome in the same record. Large multimodal requests deserve a separate route policy because a base64 image can change the memory profile long before token counting begins.

Streaming responses change the state model

The same ABI calls the plugin for each response-body chunk, which supports incremental inspection and redaction along with accounting or telemetry on a Server-Sent Events response. This creates per-stream state that must disappear when the client cancels, the connection closes, or the provider aborts.

Keep the stream context small by storing the decision identifier, policy version, a bounded inspection window, and action counters. Send final evidence during the context's log or completion phase. A plugin that accumulates the full answer inside each stream recreates response buffering and raises memory use in proportion to concurrent generations.

Full-answer rules require a deliberate trade. Buffering preserves complete context and removes token-by-token delivery. Incremental rules preserve streaming and can miss a pattern split outside the retained window. The existing AI gateway streaming response guide covers that inspection problem in detail. The Wasm implementation still needs fixtures where sensitive text crosses two SSE events, followed by cancellation before the final event.

Host functions define the real sandbox boundary

The Wasm virtual machine limits direct access to the proxy process, while the host API decides what the module can actually do. Proxy-Wasm provides access to headers and body buffers. Where the host enables them, the API can also expose metadata, logging, timers, shared data, and outbound HTTP or gRPC calls.

Treat those capabilities like service permissions. A request classifier may need body access and one policy-service destination. It has no reason to read provider credentials or rewrite the upstream authority. It also has no reason to call an arbitrary internet host. Keep secrets out of plugin configuration and environment variables. The ABI specification explicitly discourages exposing the host's environment variables.

Istio's official extensibility documentation describes Proxy-Wasm host APIs for headers and trailers. The APIs also cover metadata, outbound calls, statistics, and logging. It supports log-only and fail-open deployment modes. Fail-close mode is also available. Those choices belong in route policy. A regulated model route should fail closed when its required module fails to load or its decision dependency times out.

Module delivery is a security release

Treat each Wasm module as executable policy on the model request path. Build it in CI and attach the source revision and toolchain version. Scan the artifact before publishing it to a controlled registry. Pin the deployed artifact by digest. A mutable tag such as latest leaves the decision record unable to prove which bytes ran.

Keep a release record with these fields:

  • module name and semantic version
  • artifact digest
  • source revision and build job
  • SDK or ABI version
  • permitted host functions and outbound destinations
  • configuration hash plus route assignments
  • reviewer, approval time, canary window, and rollback digest

Record the active module digest beside every policy outcome. A policy version alone is incomplete when plugin code controls parsing or mutation before that policy runs. Six months later, a reviewer should be able to retrieve the exact module and configuration that produced a denied request.

Canary behavior before enforcement

Start the module on one test route in log-only mode. Compare its proposed decisions with the current gateway path and investigate parser disagreements. Then enforce on a small route set while retaining the previous digest for rollback.

Fault testing should cover more than a malformed module. Kill the policy dependency after request headers arrive. Exhaust the body limit one byte at a time. Send invalid UTF-8 inside JSON and close an SSE connection halfway through a sensitive pattern. Replace the module with one that fails during configuration. Confirm the declared fail behavior and the evidence produced for every case.

Test bypass by sending the same prompt directly to the model endpoint over a test network path. The Wasm plugin can govern traffic only when the request traverses its proxy. Network policy, provider credential controls, and application configuration own the direct route.

DeepInspect

DeepInspect applies identity-aware policy to authenticated HTTP traffic deliberately routed between users or agents and LLM endpoints. A Wasm filter can act as an integration point by passing the validated principal and model route to that decision layer. It can include the request content and correlation identifier before Envoy opens the upstream model path.

DeepInspect's boundary remains limited to the routed HTTP exchange. Envoy owns filter execution and module delivery. The application or identity layer establishes the caller. Network controls close direct bypass routes. DeepInspect evaluates the routed LLM request and response, then produces the decision record with the principal and policy version. The same record includes the destination, data classification, outcome, and correlation identifier.

Book a technical deep dive at deepinspect.ai.

Frequently asked questions

Should policy run inside the Wasm module or in an external service?

Small deterministic checks can run inside the module. A shared policy service fits decisions that require centrally managed rules or larger classifiers. It also fits decisions that use data that changes independently of the proxy release. The plugin can pause the request, dispatch the call, and continue or deny when the result returns. Set a timeout and route-specific failure posture.

Can a Wasm plugin replace an AI gateway?

A plugin extends a host proxy. The surrounding gateway still owns listeners, TLS, authentication, and model routing. It also owns provider credentials, connection management, and deployment operations. Treat Wasm as a policy hook inside the gateway rather than the gateway itself.

Which ABI version should the release record contain?

Record the ABI version exported by the module and supported by the host. The Proxy-Wasm project identifies v0.2.1 as its latest widely implemented specification. Test the exact host and module combination during every proxy upgrade because support and behavior can differ by host release.

How should a plugin handle policy-service failure?

Choose the failure behavior separately for each route. A route carrying regulated or restricted data should return a local denial when the policy service times out or the module fails. Lower-risk routes may use a separately approved fallback. Every fallback needs an explicit owner and expiry. It also needs an observable decision code.