BentoML Authentication: Bind Every LLM Request to a Principal
BentoML clients can attach a bearer token, while BentoML Services leave credential validation and authorization design to the deployed application or its ingress. This guide builds an authentication path for a BentoML LLM endpoint, covering route inventory, JWT validation, workload and user identity, per-route authorization, health endpoints, direct-path tests, and request evidence.

A BentoML client can place a bearer token in every HTTP request. The server side still needs to validate that credential, derive a principal, decide which endpoint and model the principal may use, and preserve evidence of the decision. A token in a client constructor proves only that a header was sent.
The distinction becomes visible when a BentoML LLM Service listens on port 3000. A developer can get a successful chat completion with the example credential while /metrics, a mounted FastAPI route, or a direct internal address follows a different policy path. Authentication needs a route inventory before it needs another token.
TL;DR
- Treat the BentoML client token as credential transport; validate it at a trusted ingress or mounted ASGI application.
- Bind each request to a workload, user or agent, tenant, role, route, and model before inference.
- Inventory inference and infrastructure endpoints separately, then test every public path for missing and invalid credentials.
- Keep BentoML deployment authentication separate from per-request LLM authorization and content policy.
- Route authenticated HTTP LLM calls through one enforcement path and record the decision without claiming control of the model host.
BentoML transports a credential to the Service
BentoML's official client documentation says SyncHTTPClient and AsyncHTTPClient accept a token, typically a JWT or API key. The client includes that token in the HTTP headers of each request. That is a useful transport mechanism. It leaves server validation and authorization to the deployment.
Write the request path on one line during design review: caller, ingress, BentoML Service, model process. Mark where TLS terminates, where the signature or bearer credential is checked, and where the principal becomes trusted context. If nobody can point to the validation step, the token is decorative.
I would reject a production review that shows only token='your_token_here' in a client screenshot. The evidence must include an anonymous request and attempts using a bad signature or expired token. Finish with a valid principal that lacks permission for the selected route.
This builds on the vLLM authentication endpoint record, but BentoML owns a different implementation surface. The BentoML client carries the token, and a BentoML deployment can mount an ASGI application beside its Service APIs.
Authentication can run at ingress or in mounted ASGI code
BentoML's ASGI integration guide states that BentoML runs Service APIs in an ASGI web-serving layer. It specifically names custom authentication and authorization as a reason to mount FastAPI or another ASGI application.
That creates two workable placements. A reverse proxy or Kubernetes ingress can validate a JWT before traffic reaches BentoML. A mounted FastAPI application can validate credentials and apply route dependencies inside the same serving process. Some deployments use both, with the ingress establishing workload identity and the application resolving finer user or tenant context.
Choose one component as the source of trusted identity. Strip caller-supplied identity headers at the public edge, then create signed or otherwise protected context after validation. The BentoML Service should receive a stable principal ID, tenant, role or delegated scope, plus the token issuer and request correlation ID. Provider keys and model-host secrets belong in the server-side secret path rather than this context.
The AI gateway architecture uses the same sequence: authenticate first, bind identity, inspect the model request, evaluate policy, then write the decision record.
Route inventory prevents partial authentication
A BentoML deployment can expose inference APIs and mounted ASGI routes alongside infrastructure endpoints such as health checks and metrics. The exact list depends on the Service and ingress configuration. Generate the route inventory from the running deployment rather than copying a documentation menu into the security packet.
For each reachable path, record the HTTP method and external path, internal target, purpose, credential requirement, accepted principal type, authorization policy, response data class, owner, plus the evidence link. Keep /livez, /readyz, and /metrics separate from model inference. A health endpoint may need anonymous access inside a cluster while remaining unreachable from the public internet.
The public allowlist should contain only routes the application needs. Reject every other path at ingress. Then test the internal address from an application subnet. A perfect public authentication rule leaves a bypass when callers can reach port 3000 directly.
BentoML's vLLM example shows a Service exposing OpenAI-compatible endpoints at /v1. That compatibility changes the client shape while leaving authentication placement and route exposure as deployment decisions.
Authorization needs the model request context
A valid credential establishes who or what called the Service. Authorization decides what that principal may do. Put the route and model into that second decision alongside operation, tenant, and data classification.
One workload may use chat completions while embeddings remain reserved for an indexing service. A support agent may summarize records for its own tenant while an administrator can run a cross-tenant quality review under a separate role. An API credential shared by both workloads erases the distinction.
Define authorization with explicit permit and deny outcomes, adding an approved transformation such as redaction where policy allows it. Record the policy version and reason code. The AI gateway versus API gateway guide explains the handoff: ordinary ingress authentication protects the Service boundary, while the AI policy decision uses verified identity and the content of the model request.
Avoid inferring user identity from a free-form header added by a browser or agent. The application should validate the user session and pass protected delegation context, or the gateway should validate a token whose claims directly identify the principal and issuer.
Failure tests belong in the release gate
Run the acceptance test against the deployed hostname and the direct Service address where network policy permits a controlled probe. Use one visible correlation ID across ingress and application logs plus the model-request decision record.
The release set should cover:
- missing, malformed, expired, and wrong-audience credentials
- a valid workload token with a forged user-role header
- an allowed principal calling a denied model or route
- a request sent directly to the BentoML port
- health and metrics paths from public and cluster networks
- authentication-service timeout during credential rotation
- a streamed response that closes before completion
A deny should stop before model inference. The record should name the trusted issuer, principal, route, model, policy version, reason, plus the upstream-send result. A 401 screenshot proves the client saw an error. The joined record proves where the request stopped.
DeepInspect
DeepInspect applies policy to authenticated HTTP traffic deliberately routed between users or agents and LLM endpoints. A BentoML deployment serving an LLM over REST can pass verified workload and user context with the selected route and model. DeepInspect can evaluate that context with the request content before forwarding a permitted call.
BentoML process security, token issuance, ingress configuration, container permissions, model files, and direct paths around the proxy remain separate controls. For traffic on the routed path, the decision record can retain the principal, tenant, role, route, model, policy version, data classification, and outcome.
Book a technical deep dive at deepinspect.ai.
Frequently asked questions
- Does the BentoML client token authenticate the server-side request by itself?
The client sends the token in an HTTP header. The deployed Service, mounted ASGI application, or ingress must validate it and decide what the resulting principal may access. Preserve both the credential-validation result and the authorization decision.
- Should authentication live in FastAPI or the ingress?
Use the ingress for consistent edge validation when several Services share an issuer and network boundary. Use mounted FastAPI dependencies when authorization depends on application-specific routes or context. If both participate, document which component creates trusted identity and how the next hop verifies it.
- Can a shared API key identify the employee behind a request?
A shared credential identifies the integration that holds it. Natural-person or agent identity requires separate verified context and a delegation link when the integration acts for a user. Record both identities on the model request.
- Do health and metrics endpoints need the same policy as inference?
They need an explicit exposure decision. Cluster health probes may use network restrictions rather than user authentication. Metrics often reveal operational data and should stay on a monitoring path. Neither route should inherit public exposure by accident.