AI usage quota enforcement: the four counters production deployments actually need

Production deployments running LLM traffic without usage quotas eventually hit one of four failure modes. A runaway agent generates a million requests in an hour. A single tenant's workload consumes the shared token budget. A misconfigured prompt template drives cost past the monthly forecast in three days. A concurrent request spike exhausts the provider's rate limit and takes down the deployment for everyone. Each failure mode needs a specific counter with fail-closed enforcement at the gateway.

The four counters are per-caller request rate, per-tenant token throughput, per-workload cost, and per-model concurrency. Provider-side rate limits are not sufficient; they enforce at the wrong granularity and fail open in the ways that matter.

I want to walk through the four counters, where each one sits in the request flow, the fail-closed behavior each one demands, and the audit fields the enforcement decisions produce.

Counter one: per-caller request rate

The per-caller request rate limits how many requests a single verified identity can send in a rolling window. The counter keys on the identity resolved at the gateway, not on the IP address or the API key. A single identity that reaches the deployment through multiple IP addresses still consumes one quota.

What this counter catches

Runaway agents. A background process that enters a retry loop and generates thousands of requests before the operator notices. Compromised credentials whose attacker uses the credential to exfiltrate data via high-volume queries. Test scripts that a developer forgot to gate behind a rate limit.

Where it enforces

At the gateway, after identity resolution and before the policy decision. A caller whose rate exceeds the limit receives a deny outcome and a 429 Too Many Requests response with a Retry-After header. The deny produces an audit record indicating the rate-limit trigger.

The fail-closed behavior

When the rate-limiter's backing store is unavailable, the gateway fails closed for the affected caller. Failing open under store unavailability would let a coordinated attack succeed during the exact window the operator was troubleshooting the store.

Counter two: per-tenant token throughput

The per-tenant token throughput limits how many prompt tokens plus completion tokens a tenant can consume in a rolling window. The counter keys on the tenant, not the individual caller. It aggregates across all callers who belong to the tenant.

What this counter catches

Tenant-level runaways. A single-tenant workload that generates a spike (a marketing campaign that quadruples chatbot traffic overnight, a batch processing job that runs full throttle instead of throttled). Cross-tenant fairness violations where one tenant consumes the shared model quota and starves other tenants.

Where it enforces

At the gateway, after tenant resolution from the caller identity. The tenant is a claim on the credential or a lookup from the identity to the tenant directory. A tenant whose throughput exceeds the limit receives a deny outcome and a message pointing to the tenant admin's quota-adjustment path.

The fail-closed behavior

The token count is estimated at request time from the prompt length; the exact count is known only after the completion. Deployments that enforce on the exact count let requests through while the counter catches up, producing overshoot. Deployments that enforce on the estimate accept some over-blocking. Fail-closed defaults to enforcement on the estimate; the trade-off is documented per tenant.

Counter three: per-workload cost

The per-workload cost limits the dollar spend a workload can accumulate in a period. The counter keys on the workload identifier (a header the caller supplies, validated against the caller's authorization). It aggregates cost across every provider the router routes to.

What this counter catches

Runaway spend. A prompt template that misuses the model (routing simple queries to the largest model, generating 4000-token responses when 200 would do). A workload that migrates from a cheap model to an expensive model without updating the cost forecast. A silent regression in a router's cost-optimized function.

Where it enforces

At the gateway, after the workload identifier is validated. The gateway maintains a running cost estimate per workload using the provider's list prices and the token counts from the audit record. Overshoots trigger a deny with a cost-attribution message.

The fail-closed behavior

Cost enforcement has a lag between the request and the provider's actual bill. The gateway's estimate is close enough for enforcement purposes, and the reconciliation to the provider's bill happens against the audit record's per-request token counts. Fail-closed defaults to the estimate; the reconciliation catches drift monthly.

Counter four: per-model concurrency

The per-model concurrency limits how many requests can be in flight against a single upstream endpoint at a time. The counter keys on the endpoint. It applies regardless of which caller or tenant the requests come from.

What this counter catches

Provider rate-limit exhaustion. A traffic spike that would otherwise exceed the provider's per-minute quota, causing every request to fail. Head-of-line blocking where a slow provider slows down the whole deployment.

Where it enforces

At the gateway, in front of the router. The router picks an upstream endpoint; the concurrency counter decides whether the endpoint has capacity. Endpoints at their concurrency limit are skipped in favor of fallback endpoints. Requests that cannot find an available endpoint queue with a bounded queue depth.

The fail-closed behavior

The queue depth is bounded. When the queue is full, new requests receive a deny with a 503 Service Unavailable response. Failing open (unbounded queue) produces memory pressure and eventual crash under load.

The audit fields quota enforcement produces

Every quota-triggered decision produces an audit record with the counter name, the current usage, the limit, the outcome, and the caller/tenant/workload/endpoint the counter keyed on. When a tenant admin asks why a specific caller was denied, the audit record answers directly. When cost accounting reconciles against the provider bill, the audit record's token counts are the source of truth.

Beyond the four counters

Advanced deployments layer more counters on top: per-role token budgets for sensitive workloads, per-time-of-day quotas for cost smoothing, per-classification limits that cap how many PHI-classified requests a tenant can send. Each additional counter answers a specific failure mode. The four above cover the base case.

The provider-side rate limits (OpenAI's per-minute token quota, Anthropic's per-minute request quota, Bedrock's per-region concurrency) still apply; the gateway's counters shape traffic to stay inside them.

DeepInspect

This is where DeepInspect's architecture matters for quota enforcement. DeepInspect maintains the four counters at the gateway and evaluates them after identity resolution and before the routing choice. Every counter reads from the verified identity, the resolved tenant, the workload identifier, and the endpoint the router intends to reach. Deny outcomes fail closed and produce audit records with the counter name and the trigger.

Every decision produces a per-decision audit record with identity, role, policy version, classification, quota counter state, decision outcome, and timestamp. Deployers reconcile cost, throughput, and rate against the audit records without pulling from provider bills for the operational answer.

Book a demo today.