LLM routing
LLM routing is the gateway function that selects which model endpoint serves a given request. The routing decision reads the route key, the caller's role, the payload classification, and the operational state of each candidate endpoint, then forwards the request to the model that satisfies the policy, latency, and cost constraints registered for that route. The selection lands in the per-decision audit record, so the organization can prove which model produced which output.
What a routing decision reads
A production routing table keys on four inputs. The route itself: a code-assist route, a summarization route, and a structured-extraction route carry different quality floors and different acceptable models. The role: a clinical user's requests may route only to endpoints covered by a Business Associate Agreement. The payload classification: a prompt carrying source code or customer records may be pinned to a specific provider region. The endpoint state: health checks and error rates drive fallback chains, so a timeout on the primary endpoint reroutes to the registered alternate instead of failing the request.
How routing becomes a governance decision
Endpoints differ in data-handling terms, retention behavior, and regional hosting, so moving a workload between models changes the compliance posture of that traffic, and a fallback that silently reroutes PHI to an endpoint without a BAA is a reportable event rather than an operational detail. Routing rules therefore belong in the same versioned policy set as block and redact rules, evaluated per request at the gateway. The audit record then names the model that actually served each request, which is the evidence a review needs when output quality or data handling is questioned later.
Related reading
- LLM routing strategies: five patterns for production, and where the policy decision constrains each one
LLM routing strategies decide which model, provider, or endpoint handles a given request. Five patterns cover most production deployments: static routing, cost-optimized routing, quality-tiered routing, latency-budgeted routing, and fallback routing. Each pattern operates on request metadata after the policy decision at the gateway has authorized the request and produced the audit record. This piece walks through the five patterns, what each optimizes for, and the constraints the gateway places on all of them.
- LLM multi-model routing: the invariants that hold when you serve traffic from more than one provider
LLM multi-model routing spreads traffic across two or more model providers so a single-vendor outage, price change, or policy shift does not stop production. The pattern is simple in principle and complicated in practice because different providers have different token formats, streaming semantics, tool-call schemas, and safety-refusal patterns. This piece walks through the six invariants that hold regardless of provider (identity resolution, classification, policy, audit, idempotency, and response normalization) and the three variances that do not (token accounting, streaming chunking, and tool-call format).
- LLM fallback routing: the retry chain that survives provider outages without leaking policy
LLM fallback routing chains a primary model to a secondary and tertiary so provider outages, rate-limit errors, and quality regressions do not cause user-visible failures. The failure modes are usually not the fallback logic itself but the boundary between the fallback chain and the policy decision that authorized the request. This piece walks through the four common triggers for fallback, the retry semantics per trigger, the authorized-endpoint constraint, and the idempotency requirements for tool-calling workloads.