← All posts

Platform & Architecture

145 posts on platform & architecture.

Stateless AI Proxy: Why the Pattern Wins for Enforcement at Scale

A stateless AI proxy is an enforcement layer for LLM traffic that does not retain per-conversation state across requests. Each request is evaluated against policy using only the inputs that arrive with the request: identity, prompt content, data classification, model destination. The architectural property matters for horizontal scaling, failure isolation, and audit independence. The piece walks through why the stateless pattern wins for enforcement-grade AI proxies, where session-state requirements live instead, and what the latency math looks like.

ai-proxyarchitectureenforcementai-gatewayengineeringai-security
Read post →

AI Security Proxy: What the Pattern Is and How It Differs from Traditional Web Proxies

An AI security proxy intercepts HTTP traffic between authenticated users or agents and LLM APIs, evaluates each request against identity-bound policy, and writes a per-decision audit record before the response returns. The pattern differs from the traditional forward proxy at four architectural points: prompt-level data classification, identity binding at the request layer, fail-closed policy evaluation, and tamper-evident audit independence. I walk through the architecture and where it fits in the 2026 enterprise AI stack.

ai-securityai-gatewayenforcementarchitectureauditai-proxy
Read post →

AI Gateway Rate Limiting: Identity-Aware Quotas at the LLM Request Boundary

AI gateway rate limiting enforces request quotas at the LLM request boundary against identity, role, model destination, and data classification. The pattern differs from a traditional API rate limit at three points: token-based budgeting that accounts for prompt and completion tokens, identity-aware quotas that bind to the caller rather than the source IP, and policy-coupled enforcement that integrates with the same gate that handles classification and audit. I walk through the quota model, the enforcement points, and where rate limiting sits relative to cost control and compliance evidence.

ai-gatewayrate-limitingengineeringai-securityenforcementcost-control
Read post →

AI Gateway TLS Termination: Why the Inspection Point Has to Decrypt the Request Body

An AI gateway terminates the outbound TLS session to the LLM provider so the inspection point can read the JSON request body in plaintext, classify the prompt content, evaluate identity-aware policy, and write a per-decision audit record. The architectural choice differs from a pass-through proxy at three points: control of the certificate chain, decryption authority over the prompt body, and re-encryption to the upstream provider with the gateway-managed identity. I walk through how the termination works, what it costs, and what the 2026 compliance set requires from the inspection point.

ai-gatewaytlsengineeringai-securityenforcementarchitecture
Read post →

AI Policy as Code: The Declarative Pattern That Makes Enforcement Auditable

AI policy as code expresses the rules that govern AI usage in a declarative configuration format checked into version control, evaluated at the AI request boundary, and versioned per decision in the audit record. The pattern differs from policy as documents at three points: machine-readable expression that the gate evaluates directly, version control that ties each decision to the policy in effect at the moment, and code review that captures the change history. I walk through what the policy actually contains, how the gate evaluates it, and how the audit record references it.

ai-policypolicy-as-codeai-securityenforcementengineeringcompliance
Read post →

The OpenAI API Gateway: Where the Inspection Point Sits Between Your Workforce and api.openai.com

An OpenAI API gateway is the inspection point your traffic to api.openai.com passes through before it reaches the model. The gateway attaches identity context, runs prompt-level classification, evaluates policy, and produces a per-decision audit record. The architecture sits between authenticated users or agents and OpenAI endpoints (chat completions, responses, embeddings, audio, batch, assistants). I walk through what the gateway intercepts, how the API surfaces map to the inspection points, and what the trade-offs are between deploying it as a SaaS-hosted proxy, a VPC-isolated proxy, or a sidecar.

ai-securityllmpolicy-enforcementinline-enforcementarchitecture
Read post →

The Anthropic API Gateway: Where the Inspection Point Sits Between Your Workforce and api.anthropic.com

An Anthropic API gateway is the inspection point HTTP traffic to api.anthropic.com passes through before it reaches Claude. The gateway attaches identity context, classifies prompt content, evaluates policy, and writes a per-decision audit record. The architecture sits between authenticated users or agents and the Anthropic endpoints (messages, batch, files, computer-use beta, prompt caching). I walk through the inspection points across each API surface, how identity attaches on top of static Anthropic API keys, and how policy enforces against Claude-specific patterns like prompt caching and the computer-use tool.

ai-securityllmpolicy-enforcementinline-enforcementarchitecture
Read post →

Bedrock API Gateway: Inspection at the AWS Bedrock Runtime Boundary

A Bedrock API gateway is the inspection point traffic to the AWS Bedrock runtime passes through before it reaches the model. The gateway attaches identity context the application supplies, runs prompt-level classification, evaluates policy, and writes a per-decision audit record. The architecture sits between callers and the InvokeModel, Converse, RetrieveAndGenerate, and agents APIs Bedrock exposes. I walk through the inspection points across each surface, how the gateway interacts with Bedrock Guardrails, and what the deployment trade-offs look like inside AWS networking.

ai-securityllmpolicy-enforcementinline-enforcementarchitecturecloud-security
Read post →

Per-Route AI Policies: Attaching Policy to the URL Path, Not the Application

Per-route AI policies attach the policy decision to the API route the request is calling, not to the application that initiated it. Different LLM endpoints carry different risk profiles. The chat-completion endpoint, the embeddings endpoint, the file-upload endpoint, the batch endpoint, the audio endpoint, and the agent action surfaces each warrant their own rules. I walk through what per-route policy looks like in practice, how route patterns express AI-specific constraints, and how the architecture composes with per-role policy and prompt-level classification at the inspection point.

policy-enforcementai-securityarchitectureinline-enforcementllm
Read post →

AI Gateway High Availability: The Failure Modes That Matter and the Topology That Survives Them

An AI gateway sits inline between the user and the LLM. When the gateway fails, the AI traffic either stops (fail closed) or bypasses the gateway (fail open). Both choices have costs. This article walks through the failure modes that matter in production, the topology patterns that survive them, and the architectural trade-offs around fail-closed vs fail-open under regulatory pressure.

ai-gatewayarchitectureai-securityinline-enforcementcloud-security
Read post →

AI Gateway Architecture for Streaming LLM Responses: Policy, Audit, Backpressure

Streaming LLM responses arrive as server-sent events or chunked HTTP, token by token, over a connection that may stay open for seconds or minutes. An AI gateway built for request-response patterns cannot enforce policy, redact sensitive content, or produce per-decision audit records on streaming traffic without re-architecting the proxy. This piece walks through the architectural changes streaming requires, the enforcement model that holds at chunk granularity, and the audit record shape that survives the inspection.

ai-gatewaystreamingai-securitypolicy-enforcementauditarchitecture
Read post →

AI Gateway Policies for Tool Use: Authorizing Function Calls at the Request Boundary

Tool use turns an LLM call into a sequence of function invocations against the application backend, the file system, third-party APIs, and other tools the model is allowed to call. Each function call has its own authorization scope and its own audit shape. An AI gateway that enforces policy on the model request alone leaves the tool invocations unauthorized. This piece walks through the architecture for authorizing tool use at the request boundary, the per-tool policy shape, and the audit record that captures the full tool-use trace.

ai-gatewaytool-usefunction-callingai-securitypolicy-enforcementaudit
Read post →