Portkey Alternatives: How to Pick a Different LLM Gateway and Observability Layer

Portkey is an LLM gateway and observability platform shipped as a closed-source product (Portkey Cloud and self-hosted enterprise deployment). The product bundles routing across 200+ providers with traces, evaluations, prompt management, guardrails, retries, fallbacks, caching, load balancing, and cost tracking on the same control plane. The platform fits teams that want one managed (or self-hosted enterprise) surface for operational LLM traffic plus the engineering team's observability. Teams that want an open-source alternative, a Kong-resident operational gateway, an observability-only layer, a Databricks-native control plane, or identity-bound policy enforcement for regulated workloads under EU AI Act Article 12 and Fannie Mae LL-2026-04 pick a different layer. I want to walk through the credible Portkey alternatives, by use case, and where each one fits.

TL;DR

Portkey bundles the LLM gateway and the observability surface on one closed-source control plane. Alternatives by use case: LiteLLM for an open-source OpenAI-SDK-compatible gateway, Kong AI Gateway for a Kong-resident operational layer, Helicone or Langfuse for observability-only deployments, MLflow AI Gateway for MLflow-anchored experimentation, Databricks AI Gateway for Databricks-resident workloads, and DeepInspect for identity-bound policy enforcement and per-decision audit records that survive regulatory review.

Use case 1: open-source OpenAI-SDK-compatible gateway

Teams that want the multi-provider gateway feature without a closed-source dependency pick an open-source alternative.

LiteLLM

LiteLLM is an open-source LLM proxy with an OpenAI-compatible API surface across 100+ providers. The proxy server handles routing, retries, fallbacks, basic key management, virtual keys with per-team budgets, and rate limits. Self-hosted deployment runs as a Python process. Teams that want an SDK-compatible multi-provider proxy with the open-source license model pick LiteLLM.

The architectural distinction versus Portkey is the surface area. Portkey covers the gateway plus the observability and evaluation surface; LiteLLM covers the gateway alone. Teams that pair LiteLLM with a separate observability product (Helicone, Langfuse) recover the equivalent surface area at the cost of operating two products instead of one.

Use case 2: Kong-resident operational gateway

Teams already running Kong as their HTTP data plane that want AI traffic management on the same operator surface pick Kong AI Gateway.

Kong AI Gateway

Kong AI Gateway is the AI-focused plugin family on the Kong data plane: multi-provider LLM routing via the AI Proxy plugin, semantic caching, prompt templates, prompt guards (regex allow and deny lists), AI-aware rate limiting, and per-consumer token attribution. The product fits teams already operating Kong that want one data plane for AI and non-AI HTTP traffic.

The architectural distinction versus Portkey is the data plane model. Kong AI Gateway runs on the open-source Kong data plane that the team operates; Portkey runs as a managed (or self-hosted enterprise) product with its own control plane. Teams with a mature Kong operations practice prefer the Kong-resident model.

Use case 3: observability-only layer

Teams that want application-side observability into LLM calls without the operational gateway features (retries, fallbacks, caching, load balancing) pick an observability-first product.

Helicone

Helicone is an open-source LLM observability platform with an async proxy and a self-hosted gateway. The dashboard exposes captured calls by user, model, route, custom property, latency, and cost. The async proxy mode does not require SDK changes for most applications.

Langfuse

Langfuse is an open-source LLM observability platform that captures traces via in-process SDKs. The trace captures multi-step spans, prompt template versions, evaluation scores, and user feedback. The platform supports prompt experimentation workflows.

The architectural distinction versus Portkey is the scope. Portkey bundles gateway and observability on one control plane; Helicone and Langfuse cover the observability layer alone. Teams that already have a multi-provider gateway in place (LiteLLM, Kong AI Gateway, MLflow AI Gateway) and need observability alongside it pick Helicone or Langfuse.

Use case 4: MLflow-anchored experimentation

Teams running LLM evaluations, prompt experimentation, and offline batch inference inside MLflow pick an MLflow-anchored gateway.

MLflow AI Gateway

MLflow AI Gateway (formerly MLflow Deployments) is an open-source MLflow component that registers LLM provider endpoints under named routes that MLflow client code calls. The MLflow tracking integration captures the call inside an MLflow run for offline review and comparison.

The architectural distinction versus Portkey is the workflow assumption. MLflow AI Gateway assumes the call is part of an MLflow run with tracking captured in MLflow's experiment surface. Portkey assumes the call is operational LLM traffic with traces and evaluations captured in the Portkey control plane.

Use case 5: Databricks-resident workloads

Teams running LLM inference primarily inside Databricks Model Serving pick the Databricks-native control surface.

Databricks AI Gateway

Databricks AI Gateway, part of Mosaic AI Gateway, sits inside Databricks Model Serving. The control plane attributes usage to Unity Catalog principals, applies AI guardrails (keyword filters, PII detection), and writes payload tables to Delta tables in Unity Catalog.

The architectural distinction versus Portkey is the identity boundary. Databricks AI Gateway assumes the caller is a Unity Catalog principal; Portkey is agnostic to the identity model and runs in front of any LLM endpoint addressable over HTTP.

Use case 6: identity-bound enforcement and regulatory audit records

Teams subject to EU AI Act Article 12, Fannie Mae LL-2026-04, HIPAA, DORA, FedRAMP, ISO 42001, or any sector regime that requires identity-bound per-decision audit records pick an enforcement-first product.

DeepInspect

DeepInspect sits at the HTTP request boundary as a separate enforcement layer. It evaluates identity-bound policy on every request, classifies prompt data against the regulated data types the organization recognizes, and commits a per-decision audit record with cryptographic integrity. The decisions are deterministic, fail-closed, and independent of the model's behavior.

The architectural distinction versus Portkey is the audit format. Portkey's operational traces satisfy the existence requirement of an audit. DeepInspect's per-decision audit records satisfy the traceability requirement that Article 12 and the Fannie Mae LL-2026-04 review apply. The record carries the natural-person identity (not the API key or virtual key alone), the policy version active at decision time, the data classification outcome, the policy decision outcome, and the cryptographic integrity signature that decouples the audit record from the application that took the action.

DeepInspect composes with Portkey by sitting in front of it for regulated workloads that also want Portkey's operational features. The composition pattern preserves Portkey's gateway and observability surface and adds the regulatory audit layer above it.

Picking between the alternatives

The right alternative depends on what the team needs from the LLM traffic layer and the audit obligation.

• Open-source gateway: LiteLLM (Python proxy) or Kong AI Gateway (Kong data plane).
• Observability-only: Helicone (proxy) or Langfuse (SDK).
• MLflow-anchored experimentation: MLflow AI Gateway.
• Databricks-resident workload: Databricks AI Gateway.
• Identity-bound policy enforcement and regulatory audit records: DeepInspect.
• Operational gateway plus regulatory audit: Portkey plus DeepInspect (composed).

Most production deployments for regulated AI workloads end up with two layers: an operational gateway (LiteLLM, Kong, Portkey, MLflow, Databricks) and a regulatory audit layer (DeepInspect). The two compose without overlap because the operational concerns and the regulatory audit obligation are different responsibilities.

DeepInspect

DeepInspect sits between calling applications and any LLM endpoint over HTTP. It evaluates identity-bound policy on every request, classifies prompt data against the regulated data types the organization recognizes, commits per-decision audit records with cryptographic integrity, and produces the record format that EU AI Act Article 12 and Fannie Mae LL-2026-04 reviewers accept. The architecture composes with Portkey by sitting in front of it, which preserves Portkey's operational and observability surface and adds the regulatory audit layer that Portkey was not designed to provide.

The composition gives organizations the operational LLM gateway plus observability they want from Portkey and the per-decision audit records they need for the workload to survive regulatory review. The audit pipeline consumes one record format regardless of which upstream provider Portkey selected for any given request.

If you are running Portkey today and the EU AI Act August 2 deadline applies to the workload, let's talk.