Semantic Caching at the LLM Gateway: What It Saves and the Four Ways It Leaks

An exact-match cache on LLM requests gets a hit rate in the low single digits, because two people asking the same question rarely type the same characters. Semantic caching embeds the incoming prompt, searches a vector store of previous prompts, and returns the stored response when cosine similarity clears a threshold. Hit rates in the 20% to 40% range are reported for support and documentation workloads, against inference latency of 500 milliseconds to several seconds and a cache lookup measured in single-digit milliseconds.

The savings are worth having, and so is an honest look at what the design costs you: a shared read path across every identity that touches the same cache namespace, and the security properties of that path are decided by two configuration values most teams set once and never revisit.

I want to walk the architecture, then the four ways it leaks, then the partitioning rules that make it safe.

The lookup path

Two knobs govern everything: the similarity threshold and the cache key. The threshold decides what counts as "the same question." The key decides who can see whose answers, and it is the one that gets under-specified.

Leak 1: The threshold is a semantic decision, not a numeric one

At 0.95 cosine similarity you get few hits and few surprises. At 0.85 the hit rate climbs and so does the class of prompts the system considers equivalent.

"What is our refund policy for enterprise customers" and "What is our refund policy for trial customers" embed close together. They differ in one token that carries the entire business meaning. Return the enterprise answer to the trial question and the cache has produced a confidently wrong response that no error rate will surface, because nothing failed.

Negation is worse. "Which regions are approved for data residency" and "Which regions are not approved for data residency" are near neighbours in most embedding spaces and opposite in meaning. Embedding models were trained to capture topic, and negation is a small perturbation on topic.

The mitigation is unglamorous: hold the threshold high, and scope caching to prompt classes where an approximate match is acceptable. Documentation lookups and boilerplate generation qualify. Anything where a single qualifier changes the answer does not.

Leak 2: Cross-tenant and cross-identity hits

A cache keyed only on the prompt embedding serves any caller whose prompt is close enough. If tenant A asked "summarise the Q3 renewal terms for Northwind" and the response was cached, tenant B asking a similar question gets tenant A's data.

This one is not theoretical. Langflow's CVE-2026-55255, a CVSS 9.9 cross-tenant IDOR that let an attacker read other tenants' embedded secrets, was actively exploited with CISA ordering federal civilian agencies to mitigate by July 10, 2026 (Help Net Security). Multi-tenant AI infrastructure with shared storage has a track record, and a semantic cache is exactly that shape.

The cache key has to include the tenant identifier and, where entitlements differ inside a tenant, the entitlement set of the caller. A key of hash(tenant_id, entitlement_group, model, params) + embedding produces a namespace per audience rather than one global pool. Hit rates drop. That is the price, and it is the correct price.

Leak 3: Personal data cached by accident

Prompts carry PII, because people paste. A cached prompt-response pair is a copy of that personal data in a second store with its own retention, its own access controls, and its own position in your records of processing.

Under UK and EU GDPR that store is a processing location that has to appear in the Article 30 register, has to honour erasure requests, and has to respect the retention period the source data carries. A subject access request asking what personal data you hold now has a second place to look, and a cache with no per-record provenance cannot answer it.

The rule I would apply: classify the prompt before the cache write, and refuse to cache anything carrying a regulated data class. Caching a support boilerplate answer is fine. Caching a response that contains a customer's account details creates a retention obligation nobody scoped.

Leak 4: The cached response skips the policy decision

This is the one that gets missed in design review, and it is the most consequential.

A response served from cache never reaches the provider, which is fine. If the cache lookup happens before policy evaluation, the response also never reaches the policy engine. A prompt that today's policy would block returns instantly from a cache populated last month under a policy version that allowed it.

The fix is ordering. Authenticate, authorise, and classify first. Consult the cache only after the request has been permitted, and record the cache-served decision with the same fields as a forwarded one, marked as a cache hit.

An audit trail with gaps wherever the cache hit is an audit trail that reports 60% to 80% of traffic and looks complete. That is a worse failure than no cache record at all, because it invites confidence.

Invalidation, briefly

Cached responses go stale in two ways. The source truth changes, so a cached answer about a pricing tier is wrong the day pricing changes. And the model version changes, so cached output no longer represents what the current model would say. Include the model identifier and generation parameters in the key, and tie cache TTL to the volatility of the underlying content rather than to a single global default. The broader treatment is in AI gateway cache invalidation.

Where I would and would not run it

I would run semantic caching on documentation lookups, FAQ deflection, classification and tagging tasks with a bounded label set, and code-comment generation. All high-volume, all tolerant of an approximate match, all rarely carrying personal data.

I would not run it on anything touching customer records, on agent loops where a cached intermediate step silently changes the trajectory, or on regulated workflows where the response has to be attributable to a specific model invocation at a specific time. The cost saving on those workloads is real and it is not worth the class of failure it buys.

Semantic caching usually gets switched on for the cost line, and the tenant key question surfaces during a security review three months later. Setting the key correctly on day one costs an afternoon.

DeepInspect

This is the ordering problem DeepInspect solves at the layer it owns. DeepInspect is a stateless proxy on the HTTP path between authenticated users or agents and any LLM, which places the identity and policy decision ahead of whatever caching sits behind it.

For every request, DeepInspect binds the call to the identity the caller presents, classifies the prompt against your own data classes, and evaluates identity-aware policy before the request proceeds, so a cache is consulted only for traffic that has already been authorised under the current policy version. Because the identity resolves at the gateway, the tenant and entitlement context needed for a correct cache key is available rather than inferred. Every decision commits a signed audit record before the response returns, on a write path the calling application never controls, which keeps the record complete across cache hits and misses alike. If your gateway serves cached responses ahead of the policy decision, book a demo today.