Anthropic API Gateway Setup: An Implementation Walkthrough for Enterprise Claude Deployments

TL;DR

• Put the gateway between the application and Anthropic, terminating application TLS, checking identity and policy, recording an audit event, then opening a separate TLS connection upstream.
• Keep the Anthropic vendor key in the gateway and give applications revocable gateway-scoped tokens, so credentials and audit ownership stay centralized.
• Inspect system prompts, messages, tool calls, tool results, cached prefixes, and every Claude SSE event, including partial tool JSON and stream error events.
• Preserve Anthropic response bodies and request IDs, enforce endpoint-specific size limits, and handle HTTP and in-stream errors without masking their status or cause.

A direct Anthropic integration sends an authenticated POST to https://api.anthropic.com/v1/messages with an x-api-key header, a model ID, and a messages array. The cleartext prompt is visible only to the application and to Anthropic. For an enterprise deployer carrying EU AI Act Article 12 obligations, NIST AI RMF MANAGE 1.3, or HIPAA, that topology does not produce a compliant audit record and does not let the deployer enforce content policy on the wire.

The gateway sits between the application and api.anthropic.com. This guide walks through the implementation with attention to Claude-specific patterns: the system prompt, tool use, prompt caching, the streaming format.

Request path

The path carries two separate TLS legs and one inspection point between them:

1. Leg A. The application opens TLS to the gateway hostname, for example https://ai-gateway.internal.example.com/v1/anthropic.
2. The inspection point. The gateway terminates leg A, resolves the caller to a verified identity, classifies the payload, evaluates policy, and writes an audit record.
3. Leg B. The gateway opens a second TLS connection to api.anthropic.com and forwards the approved request.

Pointing an application at the gateway takes two constructor arguments in the Anthropic Python SDK: base_url set to the gateway URL, and api_key set to a gateway-issued token. The rest of the SDK call stays exactly as written, including model, system, messages, and tools.

The gateway-issued token resolves to a verified caller. The Anthropic vendor key is held by the gateway, scoped to the outbound calls. An application team that leaks its gateway token leaks a credential scoped to one service and revocable in one place, which is the reason the vendor key never reaches the application host. The same split applies on the OpenAI gateway path.

Network path and IP allowlisting

Egress firewall rules on leg B need Anthropic's published ranges, and they run in two directions that carry different meanings.

• Inbound (IPv4), 160.79.104.0/23: where api.anthropic.com receives connections. Allowlist on the gateway's egress rules.
• Inbound (IPv6), 2607:6bc0::/48: the same, for IPv6 egress.
• Outbound (IPv4), 160.79.104.0/21: where Anthropic's own outbound calls originate, including MCP connector tool calls and web search.

Anthropic states these addresses will not change without notice. Five older /32 addresses in the 34.162.* space have been phased out and should be removed from any firewall rule that still carries them: 34.162.46.92, 34.162.102.82, 34.162.136.91, 34.162.142.92, and 34.162.183.95. Source: Anthropic IP addresses.

The outbound range matters for a reason that surprises most first-time deployers. Anthropic's server-side tool calls leave Anthropic's network, not the deployer's. A remote MCP server that a deployer runs and expects the gateway to protect will see traffic arriving from 160.79.104.0/21, with the gateway nowhere on that path.

Deployments on Claude Platform on AWS split this further: the inbound endpoint resolves to AWS ranges, while outbound tool calls still originate from the Anthropic ranges above.

Anthropic Messages API surface

The gateway implements the v1 surface that production Claude deployments use:

• POST /v1/messages: full classification on system, messages[].content, and tool definitions.
• POST /v1/messages with streaming: per-delta inspection on the SSE stream.
• POST /v1/messages/count_tokens: light classification (read-only intent), still produces an audit record.
• POST /v1/files: content scan on uploaded payloads.

The streaming case is where Claude-specific handling matters, and it gets its own section below.

Response format and error surface

A gateway that rewrites or swallows Anthropic's response envelope breaks client SDKs, so the proxy passes the body through unchanged and adds its own decision metadata in separate headers.

Two fields carry most of the operational weight. Every response includes a request-id header holding a value such as req_018EeWyXxfu5pfWkrYcMdjWG. Error bodies repeat the same value as a top-level request_id field. Recording the Anthropic request ID next to the gateway's own decision ID gives you a join key when a deployer opens a support ticket about a specific call.

The error body is JSON with a top-level type of error, an error object carrying type and message, and the request_id. The status codes a gateway needs to handle distinctly:

• 400 invalid_request_error: pass through, recording the request shape rather than the payload.
• 401 authentication_error: the vendor key held by the gateway is malformed, revoked, or expired, so page the platform owner.
• 402 billing_error: pass through and alert on the first occurrence, since it affects every tenant at once.
• 403 permission_error: the gateway's key lacks access to the requested resource.
• 404 not_found_error: usually a route-rewriting bug inside the gateway itself.
• 409 conflict_error: concurrent modification, safe to retry once resolved.
• 413 request_too_large: returned by Cloudflare before Anthropic's servers see it, so enforce the same limit at the gateway.
• 429 rate_limit_error: honour retry-after and attribute the limit to the calling identity in the audit record.
• 500 api_error: retry with backoff, then surface the failure together with the request ID.
• 504 timeout_error: a long non-streaming request, so route the caller to streaming or the Batches API.
• 529 overloaded_error: upstream saturation, so write the audit record and return the status unchanged.

Request size limits are enforced per endpoint: 32 MB for the Messages API and Token Counting, 256 MB for the Batch API, 500 MB for the Files API. A gateway with a lower body-size cap than the endpoint it fronts will reject valid traffic and produce a support queue nobody can explain. Set the proxy limits from this table and log every rejection. Source: Claude API errors.

System-prompt handling

Anthropic's API separates the system parameter from the messages array. The gateway inspects both. The system prompt frequently carries the most sensitive content (deployer-specific instructions, retrieved context, account-scoped data). A system prompt that carries PHI without a BAA-routed destination fails closed regardless of how clean the user message looks.

Specifically, the gateway runs the full classification chain on the system parameter before evaluating policy. The audit record names the data class of the system prompt separately from the user messages:

• data_class: phi
• data_class_components.system: phi
• data_class_components.messages: none

This separation matters in forensic review. An incident where PHI reached a non-BAA route by way of a leaky system prompt has a different remediation than one where a user pasted PHI into a chat.

Tool use

Claude's tool-use loop produces request/response pairs where the model issues tool_use blocks and the application returns tool_result blocks in the next message. Each tool-use block is a potential exfiltration vector: a model can invoke a function with arguments that name PII or PHI.

The gateway inspects the tool-use block on the outbound (model-to-application) side and the tool-result block on the inbound (application-to-model) side. The policy can:

• Block a tool call whose arguments name a customer the caller is not authorized to query.
• Redact the tool result before it returns to the model, so the model never sees fields outside the caller's scope.
• Refuse to forward a tool result whose content exceeds the prompt-side data class.

The audit record names each tool call with its arguments hash, the policy verdict, and the result hash. One entry in the tool_invocations array looks like this:

• tool_name: lookup_customer_record
• arguments_hash: sha256:8e2f0a...
• decision: pass
• result_hash: sha256:c19f3d...

Hashing rather than storing the arguments keeps the audit trail usable under a retention policy that forbids holding customer identifiers, while still proving which call was made. The wider pattern is covered in AI gateway tool-use policies.

Prompt caching

Anthropic's prompt-caching feature lets the deployer mark sections of the prompt as cacheable. The cached prefix is reused across requests at lower cost and lower latency. The gateway handles caching with a per-request inspection pass on the cached and uncached portions, with the cache-control breakpoints respected.

The audit record names the cached portion's hash separately so the forensic reader can confirm whether a request's exposure came from the cached prefix (often the system prompt) or the live extension.

The policy can refuse to cache a prefix that carries a sensitive data class. A PHI-carrying system prompt that the application marks cache_control: ephemeral is permitted only to BAA-covered destinations, and the cache is scoped to those destinations.

Streaming inspection

Setting "stream": true switches the response to server-sent events. Each event carries an SSE event name and a JSON body whose type matches it. The gateway has to parse all of them, because a proxy that only understands text deltas will corrupt tool calls and thinking blocks.

• message_start carries a Message object with empty content: open the audit record, capture the model and input token count.
• content_block_start carries the block type at an index (text, tool_use, thinking, server_tool_use): select the classifier for that block type.
• content_block_delta carries a text_delta, input_json_delta, thinking_delta, or signature_delta: feed the inspection window and forward approved chunks.
• content_block_stop closes the block at that index: parse accumulated partial JSON and run the tool-argument policy.
• message_delta carries top-level changes including stop_reason and cumulative usage: record the final token counts.
• message_stop ends the response: seal and sign the audit record.
• ping carries nothing and may appear any number of times: forward untouched.
• error delivers an error object after a 200: write the failure into the audit record.

The gateway maintains an inspection window of N tokens (default 64) and applies the completion-side classifier across the window as deltas arrive. Two Claude-specific details change how that window is built. Tool-use deltas arrive as input_json_delta events holding partial JSON strings, so the arguments only become inspectable at content_block_stop. Thinking blocks close with a signature_delta that verifies block integrity, and forwarding it altered will break the client's verification.

The token counts in message_delta usage are cumulative, not incremental. A gateway that sums them across events will bill roughly double.

The stream that ends before message_stop

"Stream ended before message_stop" is the most common Claude streaming failure in production, and the cause is documented rather than mysterious. Once the API has returned HTTP 200 and started the SSE body, later failures arrive inside the stream as an error event rather than as a status code. During periods of high load that event carries overloaded_error, the same condition that would have been a 529 before the response started.

The event is a single line of JSON: an object with "type": "error" and an inner error object naming the type and message. Nothing follows it. A client waiting for message_stop waits forever, or until its own timeout fires, which is why the symptom is reported as a hang rather than as an error.

Three handling rules follow for the gateway:

• Treat the SSE body as a second error channel with its own audit outcome. The HTTP status already read 200, so status-based logging alone records a success for a call that failed.
• Forward the error event verbatim and terminate cleanly. Anthropic's versioning policy allows new event types, and clients are expected to handle unknown types, so inventing a substitute event is riskier than passing this one through.
• For a policy block of your own, emit an error event with a policy-specific type in the same shape. Injecting a synthetic stop_reason instead corrupts a typed enum the SDKs parse, and clients will fail in ways that look like SDK bugs.

Recovery is model-dependent. On Claude 4.5 and earlier, the client resumes by sending the partial response back as the start of a new assistant message. On Claude 4.6 and later, prefilling an assistant message is rejected, so the client sends a user message containing the partial text with an instruction to continue. Tool-use and thinking blocks cannot be partially recovered at all; resume from the most recent text block. Source: Streaming messages.

A gateway that buffers the entire response before forwarding will convert every one of these mid-stream errors into a total loss, which is one reason streaming responses through an AI gateway are handled chunk by chunk.

Identity model for agentic patterns

Anthropic's tool-use loop is the most common path enterprise deployers use for agentic workflows. The gateway's identity model carries both the human principal and the agent identity through the loop:

• Outbound user-to-Claude request: subject = human principal, agent = none.
• Outbound Claude-to-tool call: subject = human principal, agent = the agent identity scoped to this conversation.
• Inbound tool-result to Claude: subject = human principal, agent = the agent identity.

Article 26 of the EU AI Act treats the agent-on-behalf model as a deployer obligation. The gateway records the principal and the agent on each call so the action lineage holds up under a regulator review.

MCP connector requests

Anthropic's MCP connector lets a Messages API request name remote MCP servers, and Claude then calls their tools without the deployer running an MCP client. The request carries the beta header anthropic-beta: mcp-client-2025-11-20 (the earlier mcp-client-2025-04-04 header is deprecated), an mcp_servers array, and a matching mcp_toolset entry in tools.

Each server definition holds four fields:

• type (required): currently only url.
• url (required): the server address, which must start with https://.
• name (required): a unique identifier, referenced by exactly one toolset.
• authorization_token (optional): an OAuth bearer token for authenticated servers.

The toolset entry names the server through mcp_server_name and controls exposure through default_config and per-tool configs, each supporting enabled and defer_loading. Setting default_config.enabled to false and enabling specific tools produces an allowlist. Leaving the default and disabling named tools produces a denylist, which is the shape most deployers want for write and delete operations.

Where the gateway can act, and where it cannot, is worth stating plainly.

The gateway sees the request declaring the servers, which means it can allowlist permitted url values, reject a request that ships an authorization_token for a server outside the approved set, and record every server named on a call. It also sees the response, where MCP results arrive as mcp_tool_use blocks (carrying id, name, server_name, and input) and mcp_tool_result blocks (carrying tool_use_id, is_error, and content). Both block types are inspectable and both belong in the audit record.

The connection from Anthropic to the MCP server runs from Anthropic's outbound range, on a leg the deployer's gateway never touches. Policy on that leg has to be enforced at the MCP server itself, through its own authorization checks. Two further constraints apply: only tool calls from the MCP specification are supported, and the server must be reachable over public HTTPS using Streamable HTTP or SSE transport, so local STDIO servers stay out of scope entirely. The connector is also excluded from zero-data-retention arrangements, which changes the answer to a procurement question a healthcare deployer will ask. Source: MCP connector. The server-side controls are covered in MCP gateway security.

Performance budget

From internal DeepInspect testing, the gateway adds 50 ms P99 across the inspection chain, the same envelope as the OpenAI gateway path. Time to first token on claude-sonnet-5 runs in the several-hundred-millisecond range in the same tests, so the inspection cost lands inside the existing budget without a user-visible regression.

Failure modes

• Anthropic 529 (overloaded). The gateway returns the 529 to the application after writing an audit record with outcome: upstream-overloaded.
• Mid-stream `overloaded_error`. The same saturation after the 200 has been sent. The audit outcome matches the 529 case even though the status line reads success.
• Policy fail-closed. Same behavior as the OpenAI path. The request returns 503, the audit record captures the reason. The fail-closed design is what makes the audit trail complete rather than best-effort.
• Streaming cut. The gateway terminates the stream and writes the partial audit record.
• Token-count drift. The gateway records both the locally counted token count and the Anthropic-reported token count for billing reconciliation, reading usage from message_delta as a cumulative figure.
• 413 at the proxy. A body-size cap below Anthropic's 32 MB Messages limit rejects valid requests. The audit record names the gateway as the rejecting party so the deployer stops filing tickets with Anthropic.

DeepInspect

DeepInspect's gateway is Anthropic-compatible at the v1 Messages level. Streaming inspection, tool-use policy, prompt-cache awareness, system-prompt data-class separation, and the chained audit format are implemented. Application teams change one line (the base_url) and the Anthropic SDK calls route through the inspection chain.

The gateway is built for the deployment profiles this guide assumes: healthcare workloads running Claude under a BAA, finance back-office workflows, and federal contractors carrying AI RMF MANAGE 1.3 obligations.

Book a technical deep dive at deepinspect.ai.