Ray Serve Authentication: Test the Token, the Route, and the Bypass
Ray 2.60 can authenticate cluster APIs with a shared token, while Ray Serve exposes application ingress over HTTP. A production release test must prove token rejection, TLS protection, route authorization, caller-context handoff, and closure of every direct-port bypass before the endpoint carries LLM traffic.

Ray Serve places an ingress deployment behind an HTTP endpoint, then routes each request to application code. Ray 2.60 also documents token authentication for Ray APIs, using one shared secret across external APIs and internal communications. A production release still needs proof that the right token reaches the right route over an encrypted path, that the application recovers caller context, and that no direct port lets traffic skip the intended controls.
I treat that proof as a release artifact. A green deployment status says the service started. It says nothing about the request that arrived without a token at 02:14 UTC.
TL;DR
- Ray Serve authentication needs a release test that covers route authorization and caller identity as well as absent or wrong tokens, encrypted transport, and direct-port bypasses.
- Ray 2.60 tokens are shared secrets sent in an HTTP header and they lack automatic expiry, so possession alone provides weak caller context.
- DeepInspect can enforce policy on authenticated LLM HTTP traffic routed through it; Ray administration and internal gRPC stay outside that boundary, along with Python execution and bypass traffic.
Ray 2.60 has two separate HTTP concerns
Ray Serve exposes its ingress deployment over HTTP. The route prefix determines where requests land, and FastAPI can supply application-level validation and dependency injection. This is the client-facing surface that may carry an LLM prompt.
Ray token authentication covers external Ray APIs and internal communications when enabled. The 2.60 documentation says the feature is available from Ray 2.52. Ray 2.59 enables it by default for local clusters, while the broader default for remote and multi-node clusters arrives in Ray 2.61. A team running 2.60 must preserve that version boundary in its evidence.
The two surfaces can sit beside each other while enforcing different things. The release record should name the public Serve route and every Ray management endpoint. It should also identify the token mode and the component that authorizes the application request.
A shared token proves possession
Ray sends the authentication token in an HTTP header. Its documentation warns that insecure HTTP exposes the value to network eavesdropping and recommends encryption for network access. The token persists for the cluster lifetime unless an operator deletes and regenerates it. It has no automatic expiry.
That gives a reviewer a precise claim: the presented request held the cluster secret. Human identity and tenant need separate evidence, as do business role and prompt authorization. If twenty services read the same token file, the token record cannot distinguish the payroll assistant from the support summarizer.
I would reject a production sign-off that stops at one successful request. The file at ~/.ray/auth_token is a concrete credential, not an identity architecture. Start the acceptance run with a missing token, then change one character in the configured value. Test the team's rotation procedure even though Ray lacks automatic expiry. Finish with a valid credential attached to a route the caller is forbidden to use.
The release test needs six checks
A release candidate should produce one retained result for each of these checks:
- Absent token: send a request without the configured secret and record the status and response body together with the receiving component and timestamp.
- Wrong token: change one character and prove the request fails before application work begins.
- Encrypted path: capture the public scheme and TLS termination point together with the certificate identity and the next hop toward the Serve ingress.
- Route authorization: use a valid credential against an allowed route and a forbidden route. Record the policy decision for both.
- Caller context: prove the application receives the user or agent identity required for the LLM decision, rather than assigning every request to the shared Ray token.
- Bypass closure: probe the ingress port and load balancer target; the node address and dashboard; and the Jobs API and any alternate service route from an unauthorized network location.
The Ray security guidance says cluster isolation must be enforced outside the Ray cluster and warns that dashboard and Jobs access can permit arbitrary code execution. The warning also covers Client access. Those management paths belong in the network test, even though they are separate from LLM request policy.
Caller context must survive the handoff
A gateway can authenticate a person or workload, yet the Serve application may receive only a bearer token and a prompt. That handoff destroys the context needed for per-user policy. The test should follow one named caller through the front door and proxy, then through the Serve ingress and model request into the audit record.
Use a synthetic identity such as finance-review-agent-17, assign it one allowed model route, and send a prompt containing a visible marker such as CASE-8421. The retained evidence should show the identity and route. It should also show the policy version and decision, with the marker at the enforcement point. A second request from the same identity to a forbidden route should produce a denial record.
This is the same post-authentication gap described in workload identity for AI gateways. Authentication establishes who presented a credential. Authorization decides what that identity may send to this LLM endpoint now.
Direct-port bypass is a release blocker
Ray Serve commonly listens on port 8000 in its documentation examples. A production topology may place a load balancer, service mesh, ingress controller, or reverse proxy in front of that port. The security property exists only when every client path crosses the intended enforcement point.
Test the node address and cluster-internal service from a network location that should lack access. Inspect Kubernetes Services, security groups, firewall rules, mesh exclusions, and health-check routes. A successful direct request turns the gateway policy into an optional path.
The release record should include the blocked connection attempt and the rule that blocked it. A diagram with a padlock icon is decoration. A packet sent to the direct address at a recorded time is evidence. The same method applies to vLLM authentication and BentoML authentication, because self-hosted model servers often acquire a second path during debugging or incident recovery.
Evidence belongs to a named release
Tie the acceptance record to the Ray version, Serve configuration digest, network-policy revision, gateway-policy version, and deployment identifier. Record the tester, UTC window, expected result, observed result, and evidence location for every check.
For release ray-serve-2026-11-13.3, list six test IDs and attach the corresponding request or connection record. Preserve failures and reruns. An overwritten screenshot erases the history a security reviewer needs.
The Ray Serve production guide covers retries, timeouts, load shedding, and deployment operations. Those checks protect availability. Authentication evidence should sit beside them in the release packet, with a distinct owner and pass condition. That separation keeps a healthy deployment from being mistaken for an authorized one.
DeepInspect
DeepInspect can evaluate authenticated client or agent requests before they reach an LLM served through Ray Serve, provided the HTTP route is deliberately forced through the proxy. It can bind the supplied identity to route and data policy, then produce a per-decision audit record for the request and response.
The boundary stays precise. Ray cluster administration, internal gRPC, Python execution, token storage, and traffic that reaches a direct Serve port remain under Ray, network, and platform controls. The release test proves those owners meet at one route without leaving a side door.
Book a technical deep dive at deepinspect.ai.