← All posts

Platform & Architecture

145 posts on platform & architecture.

Fail-closed AI gateway design: why the default failure mode is the security mode

A fail-closed AI gateway returns HTTP 503 when the policy decision point cannot reach a verdict, blocking the request rather than forwarding it. A fail-open gateway returns HTTP 200 with the upstream model response, treating the policy outage as a pass. The choice between the two postures determines whether a policy outage produces a security incident or a contemporaneous deny record. EU AI Act Article 12 and Article 26 expect the deny record. The four failure categories that test the design are policy timeout, identity provider outage, redaction engine outage, and audit write outage.

fail-closedai-gatewaypolicy-enforcementeu-ai-actaudit-logsavailability
Read post →

AI gateway observability: the metrics, traces, and logs a policy decision point should emit

An AI gateway emits four signal categories that serve four different audiences. Per-decision audit logs serve the regulator under EU AI Act Article 12. Per-request traces serve the engineering team debugging a request. Per-policy metrics serve the operations team measuring policy effects. Per-model latency histograms serve the capacity-planning team sizing the LLM provider relationship. OpenTelemetry alignment lets the four signal categories share a transport without conflating their consumers.

ai-gateway-observabilityopentelemetryaudit-logseu-ai-act-article-12policy-metricslatency-histograms
Read post →

AI policy version control: how to treat gateway policy like code

AI gateway policy that governs which users can call which models with which data lives in YAML, evolves with the organization, and carries the same regression risk as application code. Treating the policy as code means git-backed storage, semantic versioning of policy bundles, audit-log tagging of decisions with the policy version hash, blue/green policy rollout, and shadow-mode evaluation before promotion. The NIST AI RMF MAP and MANAGE functions ask the questions the version-control discipline answers.

ai-policy-version-controlpolicy-as-codenist-ai-rmfblue-green-deploymentshadow-modegitops
Read post →

AI prompt classification taxonomy: building the label set your gateway enforces against

AI prompt classification is the labelling step that produces the inputs a policy engine evaluates. The label set has to cover four dimensions: data sensitivity (PII, PHI, PCI, IP, public), intent (query, generation, code execution, agent action), risk surface (egress, lateral, instruction injection), and regulatory scope (EU AI Act high-risk, HIPAA PHI, GDPR Article 22). The policy decision joins the four dimensions against the per-user role and the per-route rule. The taxonomy is the artefact the regulator inspects when the gateway answers an audit question about why a given prompt was redacted, blocked or allowed.

ai-prompt-classificationtaxonomypolicy-engineowasp-llmdata-classificationai-policy
Read post →

AI provider rotation strategy: how to swap OpenAI for Anthropic without breaking policy or audit

AI provider rotation is the operational mechanism that lets a deployer move traffic between OpenAI, Anthropic, Google and other endpoints without breaking the policy decision or the audit trail. The mechanism requires a provider-agnostic policy model, the model identity recorded on every per-decision log, per-route routing rules that decouple the policy from the endpoint, fail-over semantics that hold the policy invariant under provider rate-limit or outage, and a documented concentration-risk posture against DORA Article 28. Rotation is the operational expression of the regulatory expectation that the deployer remains accountable across provider changes.

ai-provider-rotationmulti-providerdora-article-28concentration-riskai-gatewaypolicy-portability
Read post →

AI gateway circuit breakers: limiting blast radius when an LLM provider degrades

An AI gateway circuit breaker adapts the microservice resiliency pattern to LLM traffic. The per-provider state machine moves between closed, open, and half-open based on error rate, latency p99, and token-cost spikes. The trip thresholds, the half-open probe budget, and the breaker telemetry tie to DORA Article 19 incident reporting and produce a recovery audit trail.

ai-gatewaycircuit-breakerresiliencydora-article-19incident-responsellm-traffic
Read post →

Non-Human Identity for AI Agents: Why Service Credentials Are the Wrong Primitive

Non-human identity covers the API keys, OAuth tokens, and workload identities that authenticate services and agents to APIs. AI agents have outgrown the static-service-credential model. A single agent can act on behalf of many users, hold delegated authority that varies by task, and produce decisions that need per-action attribution. This piece walks through the four properties an NHI for AI agents must have, why static API keys fail each of them, and how identity-bound policy at the AI request boundary closes the gap.

non-human-identityai-agentsidentityengineeringnist-ai-rmf
Read post →

AI Gateway vs LLM Router: The Architectural Distinction That Matters for Enforcement

An LLM router picks the cheapest or fastest model for a given prompt. An AI gateway evaluates whether the request is permitted before any model receives it. The router optimizes cost and latency. The gateway enforces identity-bound policy and produces a per-decision audit record. This piece walks through the architectural distinction, where the two functions overlap, and why an enterprise running regulated workloads needs the gateway capability regardless of whether routing is in scope.

ai-gatewayllm-routerengineeringarchitecturepolicy-enforcement
Read post →

AI Gateway vs API Gateway: What Changes When the Payload Is a Prompt

An API gateway enforces auth, rate limits, and routing on REST and gRPC calls. An AI gateway adds prompt classification, identity-bound policy at the request payload level, and per-decision audit records. The two answer different questions about the same network position. This piece walks through the architectural distinction, the auth model differences, what an API gateway cannot enforce at the prompt layer, and where the two should sit together in production.

ai-gatewayapi-gatewayengineeringarchitecturepolicy-enforcement
Read post →

AI Gateway Deployment Patterns: Four Topologies and When Each One Fits

Where an AI gateway sits in the network topology determines what it can enforce and what it can record. Four deployment patterns dominate production: inline reverse proxy in front of the model, sidecar to the agent runtime, in-region replicas for low-latency multi-region, and dedicated tenant gateway per customer in a multi-tenant SaaS. This piece walks through the four, what each enforces, what each records, and the operational trade-offs.

ai-gatewayengineeringarchitecturedeployment-patternssre
Read post →

OWASP AISVS 1.0 Is Here: Which of the 514 Verification Requirements a Policy Gateway Enforces

OWASP released the AI Security Verification Standard (AISVS) 1.0 on June 24, 2026. The framework spans 14 chapters and 514 testable requirements, modeled after ASVS but covering prompt injection, MCP server authentication, supply chain, and runtime response handling. This article maps the gateway-relevant chapters to specific controls a stateless identity-aware policy proxy enforces, and separates them from the model, training, and supply-chain chapters that sit outside the gateway boundary.

ai-securityowaspverificationpolicy-enforcementauditarchitecture
Read post →

AI Audit Log Formats for SIEM Ingestion: Field Mapping for Splunk, Sentinel, and Chronicle

AI audit logs from a policy gateway carry fields that no traditional SIEM schema was designed for: prompt classification, response classification, agent-on-behalf-of identity, policy ID, decision outcome. The fields have to land in Splunk, Microsoft Sentinel, or Google Chronicle in a normalized form so the SOC can query across AI and non-AI signals. This article walks through the canonical AI audit field set, the mapping decisions for each major SIEM, and the pitfalls when AI evidence has to survive a regulatory inquiry months after the fact.

ai-securityaudit-logssiemsplunkcompliancearchitecture
Read post →