← All posts

Platform & Architecture

338 posts on platform & architecture.

Self-Hosted Llama Gateway Patterns: Governing an Inference Endpoint You Own

Running Llama on your own hardware with vLLM, TGI, or Ollama removes the third-party provider from the picture. It does not remove the identity, authorization, and audit problem, because a self-hosted inference server still speaks HTTP and still accepts a prompt from whoever can reach the port. This walks the gateway patterns for a self-hosted Llama endpoint: per-identity authentication on model calls, prompt classification, egress control that forces every call through one path, and a per-decision audit record of what each caller sent to the model you host.

ai-securityllm-securityidentity-and-authorizationpolicy-enforcementarchitecture
Read post →

Semantic Kernel Security Patterns: Governing What Your Kernel Sends to the Model

Microsoft Semantic Kernel orchestrates plugins, planners, and memory in your own process. When the kernel needs a model, it calls an AI service connector that reaches an LLM endpoint over HTTP. That connector call is where network security patterns attach. This walks the gateway patterns for a Semantic Kernel deployment: per-identity on the connector, classification of the prompt the kernel assembles from functions and memory, egress control, and a per-decision audit record of what the kernel asked the model.

ai-securityllm-securityagentic-aiidentity-and-authorizationpolicy-enforcement
Read post →

Pydantic AI Behind a Gateway: Routing, Identity, and What the Type System Does Not Cover

Pydantic AI gives an agent a typed output contract, typed dependencies, and validated tool signatures, which removes a real class of parsing and argument bugs. Schema validation runs after the model call and inside the process, so it covers structure rather than authorization, egress, or evidence. This walks what the type system enforces, the four gaps that remain, and how to route a Pydantic AI agent through a gateway without changing agent code.

ai-agent-securityengineeringai-gatewayllm-securitypolicy-enforcementpython
Read post →

LLM Tool Integration Security: The Trust Boundary Between a Model and the Functions It Calls

Giving a model tools converts text generation into action. The model emits a structured call, your code executes it, and the result goes back into the context window. Three trust boundaries sit inside that loop: the tool description the model reads, the arguments it emits, and the output that returns as trusted context. This walks the failure at each boundary, the validation each one needs, and where credential scoping does work no HTTP control can substitute for.

llm-securityai-agent-securityprompt-injectionpolicy-enforcementengineeringai-security
Read post →

Semantic Caching at the LLM Gateway: What It Saves and the Four Ways It Leaks

Semantic caching answers a new prompt with a stored response when the embeddings are close enough. It cuts cost and latency, and it introduces a shared read path across whatever tenants and identities share the cache namespace. This covers the architecture, the similarity threshold problem, four concrete leak paths including cross-tenant hits and stale policy decisions, and the partitioning rules that make a cache safe to run.

ai-gatewayllm-securityengineeringpolicy-enforcementai-securityarchitecture
Read post →

Identity-Aware Proxy for LLMs: Putting a Named Caller on Every Model Request

An identity-aware proxy for LLMs terminates the call between your apps and a model, reads the authenticated identity behind the request, and applies policy per caller before the prompt reaches the model. This piece covers how the pattern works at the request layer, how it differs from a classic identity-aware proxy for web apps, and why a shared API key makes the whole thing necessary.

identity-aware-proxyllm-gatewayai-securityzero-trustai-architecture
Read post →

Per-Request LLM Authorization: Deciding Access at the Call, Not the Key

Most enterprise model access is authorized once, at the API key, and never again. Per-request LLM authorization moves the decision to each individual call, evaluating the caller identity, the route, and the prompt content before the request reaches the model. This piece covers how the pattern works, why key-scoped access is too coarse for AI traffic, and how the decision stays inside the latency budget.

authorizationllm-gatewayzero-trustai-securityaccess-control
Read post →

Encrypted Reasoning Blocks Need Request-Path Policy

Research published on August 10, 2026 found that encrypted reasoning blocks could cross sessions, users, and models within provider families. Providers remediated the demonstrated extraction routes, while publicly shared traces still create a separate exposure that platform teams should govern as request-path data.

llm-securityprompt-injectionai-securitypolicy-enforcementarchitecture
Read post →

How to Log LLM Requests: The Fields a Model Call Record Needs

Logging an LLM request as if it were a generic HTTP call captures a timestamp, a status code, and a shared key, which answers almost none of the questions an incident or an audit asks. This piece defines the fields a model call record actually needs, why application code is the wrong place to write them, and what a per-decision record looks like in practice, with a concrete schema.

llm-logging'ai-audit''observability''ai-security''engineering'
Read post →

mTLS for an AI Gateway: Proving Which Service Made the Model Call

A bearer key in front of an AI gateway proves possession of a secret, not the identity of the service that holds it, and a leaked key is indistinguishable from the real caller. Mutual TLS binds each calling service or agent to a certificate the gateway verifies on every connection. This piece shows what mTLS establishes for AI traffic, what it does not, and how it feeds the authorization decision.

mtls'ai-gateway''zero-trust''authorization''engineering'
Read post →

Azure OpenAI Security: The Controls Microsoft Ships and the Layer It Leaves to You

Azure OpenAI ships real security controls: Microsoft Entra ID authentication, managed identities, private endpoints, content filters, and diagnostic logging into Azure Monitor. Those controls secure the connection and the platform. They stop at the question of whether a specific authenticated caller is permitted to send a specific prompt to a specific deployment. This piece walks the controls Azure provides, shows where they stop, and gives the identity-aware authorization pattern that closes the post-authentication gap on Azure OpenAI traffic, with the request-level detail an engineer needs.

ai-securityllm-securityidentity-and-authorizationcloud-securityinline-enforcementarchitecturepolicy-enforcement
Read post →

LLM Observability: What Traces and Token Metrics Tell You, and the Enforcement Record They Never Produce

LLM observability instruments the request path with traces, spans, token counts, latency, cost, and quality evaluations, so an engineering team can debug a slow chain or a regression. That telemetry is forensic. It tells you what happened after it happened. At an attack tempo measured in seconds it does not prevent anything, and the records it writes carry model and latency but not the caller identity, data classification, and policy decision an auditor asks for. This piece separates the debugging value of observability from the enforcement and audit layer it cannot replace.

llm-securityai-securityobservabilityinline-enforcementai-audit-loggingidentity-and-authorizationarchitecture
Read post →