← All posts

Platform & Architecture

145 posts on platform & architecture.

CrewAI Security Patterns: Governing What Your Agents Send to the Model

A CrewAI crew is Python: agents with roles, tasks, tools, and a process that runs in your own process space. Most of that orchestration sits outside a network gateway. The part a gateway governs is the outbound HTTP traffic, when an agent calls an LLM through LiteLLM or reaches an external API through an HTTP tool. This walks the security patterns for that boundary: per-agent identity on model calls, prompt classification, egress control, and an audit record of what each agent asked the model.

agentic-aiai-securityllm-securityidentity-and-authorizationpolicy-enforcement
Read post →

Azure OpenAI Gateway Setup: A Governed Proxy in Front of Your Deployments

Azure OpenAI addresses models by deployment name on a resource-specific host, authenticates with either an api-key header or a Microsoft Entra ID token, and carries an api-version query parameter on every call. This guide walks the setup for a governed gateway in front of those deployments: terminate the connection, validate the Entra ID token, bind a natural-person identity, apply per-deployment policy, constrain egress to the Azure host, and commit an audit record for each decision.

ai-securityllm-securityidentity-and-authorizationpolicy-enforcementcloud-security
Read post →

Groq API Gateway Patterns: Identity-Bound Policy at Inference Speed

Groq serves an OpenAI-compatible API on its LPU hardware, which returns completions fast enough that a slow, out-of-band control has no time to act before the response lands. That speed is the argument for inline enforcement, not against it. This walks the gateway patterns for Groq traffic: point the OpenAI-compatible base URL at the gateway, bind each call to a natural-person identity, apply per-model policy, constrain egress to the Groq host, and record each decision.

ai-securityllm-securityinline-enforcementpolicy-enforcementarchitecture
Read post →

Mistral API Gateway Patterns: One Policy Layer for Hosted and Self-Hosted

Mistral is two deployment stories under one brand: the hosted La Plateforme API at api.mistral.ai with a Bearer key, and open-weight models a team runs itself behind vLLM, TGI, or Ollama. A single identity-bound gateway can govern both, because both are HTTP endpoints. This walks the Mistral gateway patterns: bind each call to a natural-person identity, apply per-model policy, constrain egress for the hosted case, cover the self-hosted endpoint the same way, and record every decision.

ai-securityllm-securitypolicy-enforcementidentity-and-authorizationarchitecture
Read post →

Vertex AI Gateway Setup: A Governed Proxy in Front of generateContent

Vertex AI addresses models by a long resource path on a regional host, authenticates with a short-lived Google OAuth bearer token rather than a static key, and offers both the native generateContent shape and an OpenAI-compatible endpoint. This guide walks the setup for a governed gateway in front of Vertex: terminate the request, validate the Google identity, bind a natural-person identity, apply per-model policy, constrain egress to the regional host, and commit an audit record for each decision.

ai-securityllm-securityidentity-and-authorizationpolicy-enforcementcloud-security
Read post →

Self-Hosted Llama Gateway Patterns: Governing an Inference Endpoint You Own

Running Llama on your own hardware with vLLM, TGI, or Ollama removes the third-party provider from the picture. It does not remove the identity, authorization, and audit problem, because a self-hosted inference server still speaks HTTP and still accepts a prompt from whoever can reach the port. This walks the gateway patterns for a self-hosted Llama endpoint: per-identity authentication on model calls, prompt classification, egress control that forces every call through one path, and a per-decision audit record of what each caller sent to the model you host.

ai-securityllm-securityidentity-and-authorizationpolicy-enforcementarchitecture
Read post →

Semantic Kernel Security Patterns: Governing What Your Kernel Sends to the Model

Microsoft Semantic Kernel orchestrates plugins, planners, and memory in your own process. When the kernel needs a model, it calls an AI service connector that reaches an LLM endpoint over HTTP. That connector call is where network security patterns attach. This walks the gateway patterns for a Semantic Kernel deployment: per-identity on the connector, classification of the prompt the kernel assembles from functions and memory, egress control, and a per-decision audit record of what the kernel asked the model.

ai-securityllm-securityagentic-aiidentity-and-authorizationpolicy-enforcement
Read post →

Pydantic AI Behind a Gateway: Routing, Identity, and What the Type System Does Not Cover

Pydantic AI gives an agent a typed output contract, typed dependencies, and validated tool signatures, which removes a real class of parsing and argument bugs. Schema validation runs after the model call and inside the process, so it covers structure rather than authorization, egress, or evidence. This walks what the type system enforces, the four gaps that remain, and how to route a Pydantic AI agent through a gateway without changing agent code.

ai-agent-securityengineeringai-gatewayllm-securitypolicy-enforcementpython
Read post →

LLM Tool Integration Security: The Trust Boundary Between a Model and the Functions It Calls

Giving a model tools converts text generation into action. The model emits a structured call, your code executes it, and the result goes back into the context window. Three trust boundaries sit inside that loop: the tool description the model reads, the arguments it emits, and the output that returns as trusted context. This walks the failure at each boundary, the validation each one needs, and where credential scoping does work no HTTP control can substitute for.

llm-securityai-agent-securityprompt-injectionpolicy-enforcementengineeringai-security
Read post →

LiteLLM vs an AI Security Gateway: What Each One Does and Where They Compose

LiteLLM is an open-source LLM proxy that normalizes the API surface across more than 100 model providers and handles routing, retries, fallbacks, cost tracking, and basic key management. An AI security gateway sits at the same network position but answers a different question: identity-bound policy on prompt content, data classification at the request boundary, and a per-decision audit record that holds up under EU AI Act Article 12 review. The two products compose in production deployments. This piece walks through what each one does, where they overlap, and where the architectural responsibilities split.

litellmai-gatewaycomparisoninline-enforcementai-architectureaudit
Read post →

Semantic Caching at the LLM Gateway: What It Saves and the Four Ways It Leaks

Semantic caching answers a new prompt with a stored response when the embeddings are close enough. It cuts cost and latency, and it introduces a shared read path across whatever tenants and identities share the cache namespace. This covers the architecture, the similarity threshold problem, four concrete leak paths including cross-tenant hits and stale policy decisions, and the partitioning rules that make a cache safe to run.

ai-gatewayllm-securityengineeringpolicy-enforcementai-securityarchitecture
Read post →

Identity-Aware Proxy for LLMs: Putting a Named Caller on Every Model Request

An identity-aware proxy for LLMs terminates the call between your apps and a model, reads the authenticated identity behind the request, and applies policy per caller before the prompt reaches the model. This piece covers how the pattern works at the request layer, how it differs from a classic identity-aware proxy for web apps, and why a shared API key makes the whole thing necessary.

identity-aware-proxyllm-gatewayai-securityzero-trustai-architecture
Read post →