← Blog

AI Gateway on Istio Ambient: Route Model Traffic Through a Verifiable Egress Path

Parminder Singh
Parminder Singh··7 min read
Summarize with AI

Istio Ambient gives Kubernetes teams a sidecar-free L4 overlay and optional L7 waypoints. An AI gateway deployment needs both a verifiable egress route and an HTTP inspection point that can read model requests. This guide separates ztunnel, waypoint, TLS, identity, bypass prevention, and AI policy into an implementation plan with acceptance tests.

Platform & Architectureai-securityinline-enforcementpolicy-enforcementkubernetesarchitecture
AI Gateway on Istio Ambient: Route Model Traffic Through a Verifiable Egress Path

Istio Ambient splits the mesh data plane into a per-node L4 proxy and an optional Envoy waypoint for L7 processing. That split changes the AI gateway design. Ztunnel can authenticate workloads and carry encrypted traffic, while the waypoint can route HTTP and enforce L7 policy. A prompt sent inside end-to-end HTTPS still reaches those components as encrypted bytes. The deployment therefore needs a deliberate HTTP inspection point, plus proof that model traffic cannot take a direct path around it.

I want to make that topology concrete. A diagram with one arrow through a waypoint proves very little. The useful artifact is a route test that names the source workload, provider host, identity context, and inspection decision. It also records the failed bypass attempt.

TL;DR

  • Istio Ambient's ztunnel provides L4 transport security and workload identity, but it does not parse HTTP headers or prompt bodies.
  • A waypoint adds L7 routing and policy. External LLM traffic also needs a ServiceEntry and a deliberate HTTP inspection point.
  • End-to-end HTTPS hides the prompt JSON from generic mesh proxies, so the AI gateway must terminate the application-side request and create a new TLS connection upstream.
  • Acceptance requires an allowed call and a policy denial. A direct-provider bypass must also fail.

Ambient mode separates transport from HTTP policy

The Istio Ambient overview describes two data-plane layers. Ztunnel runs once per node and handles L3 and L4 routing, mTLS, authentication, L4 authorization, and telemetry. Istio states that ztunnel does not terminate workload HTTP traffic or parse HTTP headers. Its job is secure transport through the HBONE tunnel.

A waypoint is an Envoy deployment outside application pods. It supplies L7 authorization and HTTP telemetry. It also provides traffic management for enrolled resources. Ztunnel can establish which workload service account opened a connection. The waypoint can apply HTTP route policy. The identity-aware AI gateway still needs the human or agent identity supplied by the application, because a Kubernetes service account identifies the workload rather than the person behind one model request.

Treating ztunnel as a prompt inspection control is an architectural error. It never sees the JSON messages array in a form it can classify. I would reject any design review that labels ambient enrollment alone as AI policy enforcement.

The egress waypoint creates a verifiable provider route

Istio's current ambient egress gateway guide uses a dedicated namespace and a waypoint. It also uses a ServiceEntry for the external host. When that namespace is enrolled, ztunnel routes mesh traffic for the ServiceEntry through the waypoint. The guide verifies traversal with the waypoint's upstream_rq_total counter.

For model traffic, create one reviewed ServiceEntry per approved provider host or bounded host group. Keep those objects in a dedicated namespace such as istio-egress, then attach the waypoint to that namespace. Record the owner and provider purpose beside each entry. An old ServiceEntry for a retired model endpoint is an unreviewed outbound route, even when every packet still crosses Istio.

Enrollment supplies a route through the waypoint, but Istio warns that this setting by itself does not prevent a direct path when the waypoint is unavailable. Its documented fix is an AuthorizationPolicy that allows only the waypoint identity, enforced at L4 by ztunnel. Network controls should also deny workload egress to provider addresses except through the approved path. The AI egress control then has a route it can test rather than a policy diagram it has to trust.

TLS determines which layer can inspect the prompt

Istio distinguishes HTTP from TLS traffic in its traffic-routing documentation. HTTP exposes the host, path, headers, and query parameters to Envoy. TLS exposes much less, commonly the destination and SNI needed for routing. An application that opens HTTPS directly to a model provider keeps the prompt body encrypted across a generic pass-through waypoint.

The AI gateway must become the application-side HTTP destination. Configure the model client to call an internal gateway service and carry the originating identity in an authenticated application token. Let the gateway establish a separate HTTPS connection to the provider. Ambient mTLS protects the in-mesh hop. The gateway terminates that request and evaluates the visible JSON body. It then sends an approved request upstream under its provider credential.

Istio also documents TLS origination at an ambient egress waypoint. In that pattern, a workload sends HTTP to the waypoint over ambient mTLS, and the waypoint starts HTTPS to the external host. That pattern makes HTTP available at the waypoint, but it does not create prompt classification or per-request user authorization. It also does not create an independent AI decision record by itself. Those functions need an AI-aware processor or gateway on the visible HTTP path. The TLS termination analysis covers that inspection boundary in more detail.

Identity needs two separate assertions

Ambient provides the workload identity for the mesh connection. The application provides the actor identity used for AI policy. Conflating them creates a clean-looking audit record that names a Kubernetes service account for every request and leaves the actual analyst or agent unresolved.

The request arriving at the AI gateway should carry a token that the gateway verifies, with the subject, issuer, audience, expiry, and approved role claims. The gateway should bind those claims to the workload identity observed on the in-mesh connection. A request from the expected service account with a token issued for another audience should fail. So should a valid user token arriving through an unapproved workload.

The resulting decision record needs the actor subject and workload identity, along with the provider host, model route, prompt classification, policy version, outcome, UTC timestamp, and correlation identifier. This is the point where zero trust AI becomes specific: two authenticated facts enter one decision, and neither one substitutes for authorization of the prompt being sent.

A sticky note saying svc-ai-client on the architecture board is not human attribution. Keep that distinction visible in the deployment review.

Policy ownership stays split across the path

Istio owns mesh enrollment, workload mTLS, ServiceEntry discovery, waypoint routing, L4 bypass policy, and the network evidence that traffic traversed the expected proxy. The AI gateway owns prompt parsing, classification, actor-aware route authorization, redaction or denial, and the per-decision record. The provider owns its API credential checks and model service.

During failure handling, a workload should lose provider access when ztunnel cannot establish the approved path. If the AI gateway cannot verify identity or load its policy, the model request should fail closed. A provider timeout is a delivery failure and belongs in route telemetry. It should not be rewritten as a policy denial.

Direct calls around the gateway remain invisible to its inspection logic. Istio and the surrounding network must make that bypass fail. Local prompt construction, files read before the request, tool execution after the response, and stolen credentials keep their existing owners. The authenticated HTTP request and response are the AI gateway's stop line.

Acceptance tests prove the route and the decision

Run the test from a named workload in an ambient-enrolled namespace. Use synthetic data and a test identity so the evidence can be retained without creating a second data-handling problem.

  • Allowed route: send a permitted prompt through the internal AI gateway. Preserve the gateway correlation identifier, the allow decision, the destination model, and evidence that the egress waypoint forwarded the upstream request.
  • Policy denial: change the actor role or insert a synthetic restricted value. Expect a gateway denial and verify that no matching upstream provider request occurred.
  • Direct bypass: call the provider host from the workload without using the internal gateway. Expect the connection or authorization step to fail under the Istio and network policy.
  • Identity mismatch: present a valid test token through an unapproved workload identity. Expect denial before the request reaches the provider.
  • Dependency failure: make the policy service unavailable. Expect the gateway to fail closed and preserve a decision or operational error record under the same test correlation identifier.

Place one allowed request beside one policy denial. Add one failed direct call in the adjacent terminal output. Pair those lines with the ServiceEntry, AuthorizationPolicy, gateway policy version, and decision records.

DeepInspect

DeepInspect can serve as the internal HTTP destination for model clients in this design. The application sends an authenticated request to DeepInspect through the ambient mesh. DeepInspect verifies the supplied actor context, classifies the prompt, evaluates per-role and per-route policy, and records the decision before any approved request is forwarded to the LLM endpoint. That decision may allow or deny the request, with redaction where policy requires it.

DeepInspect stays within the authenticated HTTP request and response path it can inspect. A local file read or tool execution remains outside that boundary. The same applies to a direct connection around the approved route, and each needs its own control owner.

Book a technical deep dive at deepinspect.ai.

Frequently asked questions

Does Istio Ambient replace an AI gateway?

Istio Ambient supplies mesh transport, workload identity, routing, and policy enforcement at L4 or L7 depending on the data-plane layer. An AI gateway parses model-specific HTTP payloads, applies actor-aware data and route policy, and records the decision. The two layers compose on the same path, with different owners and evidence.

Can ztunnel inspect prompts?

No. Istio documents ztunnel as an L3 and L4 proxy that does not terminate workload HTTP traffic or parse HTTP headers. It can secure and authorize the connection. Prompt inspection requires an HTTP termination point that can read the model request body.

Is a waypoint enough for external LLM traffic?

A waypoint can route external traffic represented by a ServiceEntry and apply L7 policy when HTTP is visible. Point the model client at an internal AI gateway when the policy needs prompt classification and per-request actor authorization. The egress waypoint then proves and restricts the gateway's upstream provider route.

Which identity should the audit record use?

The audit record should keep both identities. Ambient mesh establishes the workload identity, commonly a Kubernetes service account. An application token establishes the human or agent subject. The AI policy decision should bind both identities to the provider route and policy version used for that request.

How do we prove that workloads cannot bypass the gateway?

Test a direct provider call from an enrolled workload and require it to fail. Retain the relevant Istio AuthorizationPolicy, network egress rule, waypoint traversal evidence, and gateway decision record.