← All posts

Platform & Architecture

227 posts on platform & architecture.

Semantic Caching at the LLM Gateway: What It Saves and the Four Ways It Leaks

Semantic caching answers a new prompt with a stored response when the embeddings are close enough. It cuts cost and latency, and it introduces a shared read path across whatever tenants and identities share the cache namespace. This covers the architecture, the similarity threshold problem, four concrete leak paths including cross-tenant hits and stale policy decisions, and the partitioning rules that make a cache safe to run.

ai-gatewayllm-securityengineeringpolicy-enforcementai-securityarchitecture
Read post →

Identity-Aware Proxy for LLMs: Putting a Named Caller on Every Model Request

An identity-aware proxy for LLMs terminates the call between your apps and a model, reads the authenticated identity behind the request, and applies policy per caller before the prompt reaches the model. This piece covers how the pattern works at the request layer, how it differs from a classic identity-aware proxy for web apps, and why a shared API key makes the whole thing necessary.

identity-aware-proxyllm-gatewayai-securityzero-trustai-architecture
Read post →

Per-Request LLM Authorization: Deciding Access at the Call, Not the Key

Most enterprise model access is authorized once, at the API key, and never again. Per-request LLM authorization moves the decision to each individual call, evaluating the caller identity, the route, and the prompt content before the request reaches the model. This piece covers how the pattern works, why key-scoped access is too coarse for AI traffic, and how the decision stays inside the latency budget.

authorizationllm-gatewayzero-trustai-securityaccess-control
Read post →

Encrypted Reasoning Blocks Need Request-Path Policy

Research published on August 10, 2026 found that encrypted reasoning blocks could cross sessions, users, and models within provider families. Providers remediated the demonstrated extraction routes, while publicly shared traces still create a separate exposure that platform teams should govern as request-path data.

llm-securityprompt-injectionai-securitypolicy-enforcementarchitecture
Read post →

How to Log LLM Requests: The Fields a Model Call Record Needs

Logging an LLM request as if it were a generic HTTP call captures a timestamp, a status code, and a shared key, which answers almost none of the questions an incident or an audit asks. This piece defines the fields a model call record actually needs, why application code is the wrong place to write them, and what a per-decision record looks like in practice, with a concrete schema.

llm-logging'ai-audit''observability''ai-security''engineering'
Read post →

mTLS for an AI Gateway: Proving Which Service Made the Model Call

A bearer key in front of an AI gateway proves possession of a secret, not the identity of the service that holds it, and a leaked key is indistinguishable from the real caller. Mutual TLS binds each calling service or agent to a certificate the gateway verifies on every connection. This piece shows what mTLS establishes for AI traffic, what it does not, and how it feeds the authorization decision.

mtls'ai-gateway''zero-trust''authorization''engineering'
Read post →

Azure OpenAI Security: The Controls Microsoft Ships and the Layer It Leaves to You

Azure OpenAI ships real security controls: Microsoft Entra ID authentication, managed identities, private endpoints, content filters, and diagnostic logging into Azure Monitor. Those controls secure the connection and the platform. They stop at the question of whether a specific authenticated caller is permitted to send a specific prompt to a specific deployment. This piece walks the controls Azure provides, shows where they stop, and gives the identity-aware authorization pattern that closes the post-authentication gap on Azure OpenAI traffic, with the request-level detail an engineer needs.

ai-securityllm-securityidentity-and-authorizationcloud-securityinline-enforcementarchitecturepolicy-enforcement
Read post →

LLM Observability: What Traces and Token Metrics Tell You, and the Enforcement Record They Never Produce

LLM observability instruments the request path with traces, spans, token counts, latency, cost, and quality evaluations, so an engineering team can debug a slow chain or a regression. That telemetry is forensic. It tells you what happened after it happened. At an attack tempo measured in seconds it does not prevent anything, and the records it writes carry model and latency but not the caller identity, data classification, and policy decision an auditor asks for. This piece separates the debugging value of observability from the enforcement and audit layer it cannot replace.

llm-securityai-securityobservabilityinline-enforcementai-audit-loggingidentity-and-authorizationarchitecture
Read post →

OWASP Rebuilt Its LLM Top 10 Around 6,639 Real Incidents, and Excessive Agency Climbed to Third

OWASP published the 2026 edition of its Top 10 for LLM Applications on 3 August 2026. For the first time the ranking was set by two inputs rather than one: expert consensus carried 75% of the weight and 6,639 documented real-world incidents carried the remaining 25%. Excessive Agency climbed from sixth place to third. This walks the three entries that live on the request and response path, and names the entries a policy gateway has no claim on.

llm-securityai-securitypolicy-enforcementarchitectureagentic-ai
Read post →

Ghostjacking: Tenet Turned Blocked Requests Into Agent Instructions at DEF CON 34

Tenet Security presented Ghostjacking at DEF CON 34 on August 9, 2026. An attacker sends a request designed to be blocked, the firewall logs the payload verbatim, and an AI agent asked to review that log follows the attacker''s text as instruction. In the demonstrated chain the agent rewrote DNS records to point at attacker infrastructure. This piece walks the mechanism step by step and separates the observability fixes from the two controls that live on the AI request path.

llm-securityagentic-aiprompt-injectionai-securitypolicy-enforcementidentity-and-authorization
Read post →

Azure MCP Server SSRF and Credential Relay: Six Services That Carry a Managed Identity Token Wherever the Request Points

At DEF CON 34 Cloud Village on August 8, 2026, Marios Gyftos and Chrysostomos Manousis presented credential relay findings across six Azure services, including Azure AI Foundry, Azure AI Speech, Azure MCP Servers, AKS MCP and API Management. The root cause is one sentence long: the credential attached to an outbound request and the destination of that request are resolved independently, and nothing checks that they belong together. This piece walks the mechanism, separates the part Microsoft owns from the part a policy gateway owns, and sets out the record an incident reviewer needs.

ai-securitymcpazuressrfcredential-relaypolicy-enforcementllm-security
Read post →