← All posts

Platform & Architecture

338 posts on platform & architecture.

AI policy version control: how to treat gateway policy like code

AI gateway policy that governs which users can call which models with which data lives in YAML, evolves with the organization, and carries the same regression risk as application code. Treating the policy as code means git-backed storage, semantic versioning of policy bundles, audit-log tagging of decisions with the policy version hash, blue/green policy rollout, and shadow-mode evaluation before promotion. The NIST AI RMF MAP and MANAGE functions ask the questions the version-control discipline answers.

ai-policy-version-controlpolicy-as-codenist-ai-rmfblue-green-deploymentshadow-modegitops
Read post →

AI prompt classification taxonomy: building the label set your gateway enforces against

AI prompt classification is the labelling step that produces the inputs a policy engine evaluates. The label set has to cover four dimensions: data sensitivity (PII, PHI, PCI, IP, public), intent (query, generation, code execution, agent action), risk surface (egress, lateral, instruction injection), and regulatory scope (EU AI Act high-risk, HIPAA PHI, GDPR Article 22). The policy decision joins the four dimensions against the per-user role and the per-route rule. The taxonomy is the artefact the regulator inspects when the gateway answers an audit question about why a given prompt was redacted, blocked or allowed.

ai-prompt-classificationtaxonomypolicy-engineowasp-llmdata-classificationai-policy
Read post →

AI provider rotation strategy: how to swap OpenAI for Anthropic without breaking policy or audit

AI provider rotation is the operational mechanism that lets a deployer move traffic between OpenAI, Anthropic, Google and other endpoints without breaking the policy decision or the audit trail. The mechanism requires a provider-agnostic policy model, the model identity recorded on every per-decision log, per-route routing rules that decouple the policy from the endpoint, fail-over semantics that hold the policy invariant under provider rate-limit or outage, and a documented concentration-risk posture against DORA Article 28. Rotation is the operational expression of the regulatory expectation that the deployer remains accountable across provider changes.

ai-provider-rotationmulti-providerdora-article-28concentration-riskai-gatewaypolicy-portability
Read post →

AI gateway circuit breakers: limiting blast radius when an LLM provider degrades

An AI gateway circuit breaker adapts the microservice resiliency pattern to LLM traffic. The per-provider state machine moves between closed, open, and half-open based on error rate, latency p99, and token-cost spikes. The trip thresholds, the half-open probe budget, and the breaker telemetry tie to DORA Article 19 incident reporting and produce a recovery audit trail.

ai-gatewaycircuit-breakerresiliencydora-article-19incident-responsellm-traffic
Read post →

Non-Human Identity for AI Agents: Why Service Credentials Are the Wrong Primitive

Non-human identity covers the API keys, OAuth tokens, and workload identities that authenticate services and agents to APIs. AI agents have outgrown the static-service-credential model. A single agent can act on behalf of many users, hold delegated authority that varies by task, and produce decisions that need per-action attribution. This piece walks through the four properties an NHI for AI agents must have, why static API keys fail each of them, and how identity-bound policy at the AI request boundary closes the gap.

non-human-identityai-agentsidentityengineeringnist-ai-rmf
Read post →

AI Gateway Deployment Patterns: Four Topologies and When Each One Fits

Where an AI gateway sits in the network topology determines what it can enforce and what it can record. Four deployment patterns dominate production: inline reverse proxy in front of the model, sidecar to the agent runtime, in-region replicas for low-latency multi-region, and dedicated tenant gateway per customer in a multi-tenant SaaS. This piece walks through the four, what each enforces, what each records, and the operational trade-offs.

ai-gatewayengineeringarchitecturedeployment-patternssre
Read post →

AI Audit Log Formats for SIEM Ingestion: Field Mapping for Splunk, Sentinel, and Chronicle

AI audit logs from a policy gateway carry fields that no traditional SIEM schema was designed for: prompt classification, response classification, agent-on-behalf-of identity, policy ID, decision outcome. The fields have to land in Splunk, Microsoft Sentinel, or Google Chronicle in a normalized form so the SOC can query across AI and non-AI signals. This article walks through the canonical AI audit field set, the mapping decisions for each major SIEM, and the pitfalls when AI evidence has to survive a regulatory inquiry months after the fact.

ai-securityaudit-logssiemsplunkcompliancearchitecture
Read post →

MCP Server Authentication for Enterprise Deployments: Identity, Authorization, and the Boundary Question

Model Context Protocol (MCP) servers have moved from developer-tool integrations to production agent backends inside enterprises. The authentication and authorization model for MCP traffic differs from REST-API authentication in two specific ways that matter at enterprise scale: the principal acting on the MCP server can be an agent on behalf of a human, and the tool invocations have to carry the propagation chain. This article walks through the MCP authentication model, the transport-boundary distinction, and the gateway-layer controls that produce a usable audit trail.

ai-securitymcpauthenticationagent-securityarchitecturepolicy-enforcement
Read post →

AI Gateway Canary Deployment: Patterns for Rolling Policy and Model Changes Safely

Canary deployment for an AI gateway covers two distinct change types: model routing changes (a new provider, a new model version, a new model entirely) and policy changes (a new redaction rule, a new tool allowlist, a new rate-limit threshold). Each change type has different risk characteristics and different rollback triggers. The canary pattern at the gateway differs from a classic application canary because the unit of traffic is identity-bound and the failure modes include silent drift in model behavior. This article walks through the canary architecture for an AI gateway, the metrics that drive the rollout, and the rollback conditions that have to be wired in before the canary starts.

ai-gatewaycanary-deploymentdeployment-patternsrelease-managementai-policy-enforcementplatform-engineering
Read post →

AI Gateway Rate Limiting by Identity: Why Per-Key Limits Fail in Production

AI gateway rate limiting that uses the API key as the limit boundary fails in three production patterns: shared service accounts, agent fan-out, and cost runaway from a single high-volume identity. The fix is to limit per verified identity, where identity is the authenticated principal extracted from the request context, not the API key in the header. This article walks the failure modes, the architecture that fixes them, the data model the gateway needs, and the operational tradeoffs of identity-bound limits versus simpler per-key approaches.

ai-gatewayrate-limitingidentityengineeringpolicy-enforcementagentic-ai
Read post →

AI Gateway Fail-Open vs Fail-Closed: The Decision That Shapes Your Audit Trail

An AI gateway that sits inline between authenticated callers and the LLMs they use has to answer a structural question. When the gateway cannot reach the policy decision (the policy engine is down, the identity service is unreachable, a configuration cannot be loaded), does the request go through (fail-open) or get refused (fail-closed)? The answer shapes the audit trail, the regulatory posture, and the production behavior under degraded conditions. This article walks the tradeoffs, the cases where each mode is appropriate, the data-driven defaults, and the operational patterns that hold up under audit.

ai-gatewayfail-closedreliabilityengineeringpolicy-enforcementaudit-logging
Read post →

AI Gateway Rollback Strategy: How to Revert a Policy or Model Change Without Breaking the Audit Trail

A bad policy change or a broken model upgrade at the AI gateway has to be reverted fast. The rollback is the high-availability move that prevents a small problem from becoming a service-wide outage. The rollback also has to preserve the audit trail, because the regulatory record of "what policy was in effect when" survives the rollback. This article walks the rollback patterns that work at the gateway layer, the failure modes that catch teams off guard, the integrity controls that keep the audit record consistent across the revert, and the operational drill that proves the rollback works before it has to.

ai-gatewayrollbackdeploymentpolicy-versioningaudit-logginghigh-availability
Read post →