TL;DR:

  • Portkey is an open-source AI gateway that gives you routing, fallbacks, load balancing, semantic caching, and full observability across LLM providers — without changing your existing OpenAI-compatible client code
  • It handles the reliability concerns that matter in production: what happens when a provider is rate-limited, which model to use for which request type, how to track costs and latency per team or feature
  • Self-hostable as a Docker container or available as a managed cloud service; either way the integration is a one-line base URL change

Running a single LLM call in a demo is trivial. Running LLM calls reliably in production — across multiple providers, with cost controls, with visibility into what’s actually happening — is an infrastructure problem. Portkey is one approach to solving it at the gateway layer, rather than inside application code.

What Portkey Does

Portkey acts as a proxy between your application and one or more LLM providers. Your code calls Portkey’s API (which is OpenAI-compatible), and Portkey handles routing, retrying, falling back, and logging before the request hits the actual model.

The core capabilities:

Fallbacks — if your primary provider returns an error, rate limit, or timeout, Portkey automatically retries with a configured fallback provider or model. You define the fallback chain in a config; your application code doesn’t need to handle provider errors directly.

Load balancing — distribute requests across multiple API keys or providers by weight. Useful for teams that have multiple OpenAI org accounts, or want to split traffic between GPT-4o and Claude for cost or performance reasons.

Semantic caching — cache responses by semantic similarity, not exact string match. A cached answer to “what’s the capital of France?” serves a follow-up question “what city is the capital of France?” without hitting the model. Reduces costs significantly for use cases with repetitive or similar queries.

Guardrails — input and output filters that run before/after the LLM call. Block prompt injection attempts, PII in outputs, or topic categories your application should never engage with. Portkey ships built-in guardrail checks and supports custom ones.

Observability — every request is logged with latency, token counts, cost, provider, model, and metadata you attach (user ID, feature name, team). You get dashboards, per-dimension breakdowns, and the ability to trace individual requests through multi-step chains.

Virtual keys — manage LLM provider API keys centrally in Portkey rather than distributing them across services. Your application uses a Portkey virtual key; the real provider key never leaves the gateway. Rotation, revocation, and rate limiting per virtual key without touching application deployments.

Installation and Setup

Portkey is available as a managed service at portkey.ai or self-hosted via Docker:

docker run -d \
  -p 8787:8787 \
  -e PORTKEY_API_KEY=your_portkey_key \
  portkeyai/gateway:latest

For the managed service, you get the gateway at api.portkey.ai. For self-hosted, point to your container.

Adding Portkey to an existing OpenAI client is a one-line change:

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

client = OpenAI(
    api_key="your_openai_key",
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key="your_portkey_api_key",
        virtual_key="openai-virtual-key"
    )
)

# Existing OpenAI calls work unchanged
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)

No other code changes. The response format is identical to the OpenAI SDK response.

Configs: Defining Routing Logic

Portkey’s routing behaviour is defined in configs — JSON objects that specify how requests should be handled. Configs are stored in Portkey and referenced by ID, or passed inline per-request.

A fallback config that tries GPT-4o first, then Claude Sonnet:

{
  "strategy": {
    "mode": "fallback"
  },
  "targets": [
    {
      "virtual_key": "openai-virtual-key",
      "model": "gpt-4o"
    },
    {
      "virtual_key": "anthropic-virtual-key",
      "model": "claude-sonnet-4-6"
    }
  ]
}

A load balancing config distributing 70% to GPT-4o and 30% to Claude:

{
  "strategy": {
    "mode": "loadbalance"
  },
  "targets": [
    {
      "virtual_key": "openai-virtual-key",
      "model": "gpt-4o",
      "weight": 0.7
    },
    {
      "virtual_key": "anthropic-virtual-key",
      "model": "claude-sonnet-4-6",
      "weight": 0.3
    }
  ]
}

Configs can also specify retry behaviour, timeouts, and cache settings per target.

Metadata and Cost Attribution

One of the more practically useful features for teams is metadata tagging. Attach arbitrary key-value metadata to requests and Portkey aggregates costs and usage by those dimensions:

headers = createHeaders(
    api_key="your_portkey_api_key",
    virtual_key="openai-virtual-key",
    metadata={
        "user_id": "u_12345",
        "feature": "invoice_extraction",
        "team": "finance-automation"
    }
)

In the Portkey dashboard, you can then see cost, token usage, and latency broken down by feature, team, or user — which matters when you have multiple product areas sharing LLM infrastructure and need to understand where spend is going.

Prompt Management

Portkey includes a prompt library for storing, versioning, and A/B testing prompts outside of application code. You reference a prompt by ID:

response = portkey.prompts.completions.create(
    prompt_id="invoice-extraction-v3",
    variables={"document_text": invoice_text}
)

Prompt versions are managed in Portkey’s UI. Rolling back to a previous prompt version is a UI action, not a code deployment. For teams where prompt iteration is frequent, this separates the prompt lifecycle from the application release cycle.

Observability and Tracing

Every Portkey request gets a trace ID. For multi-step chains (retrieval → reranking → generation, for example), you can group related calls under a single trace using the trace_id metadata field:

import uuid
trace_id = str(uuid.uuid4())

# Step 1: retrieval query
retrieval_headers = createHeaders(
    api_key="your_portkey_api_key",
    virtual_key="openai-virtual-key",
    trace_id=trace_id,
    metadata={"step": "retrieval"}
)

# Step 2: generation
generation_headers = createHeaders(
    api_key="your_portkey_api_key",
    virtual_key="openai-virtual-key",
    trace_id=trace_id,
    metadata={"step": "generation"}
)

Both calls appear as a single trace in the dashboard, with latency, cost, and token usage per step. This matters for debugging: when a complex chain produces a bad output, you can see exactly which step contributed to the cost and whether any provider returned a non-200 that triggered a fallback.

When Portkey Makes Sense

Portkey adds a network hop. For latency-sensitive applications where every millisecond counts, that matters. For most production LLM applications — where model latency dominates over gateway overhead — it doesn’t.

Use Portkey (or a similar gateway) when:

  • You’re calling more than one LLM provider and want a single integration surface
  • You need cost attribution across teams, features, or users
  • You want fallback behaviour without building it into application code
  • You’re managing API keys across multiple services and want centralised control
  • You need a compliance-ready audit trail of all model inputs and outputs

Build directly to provider APIs when:

  • You’re using a single provider with no plans to switch
  • You’re in a latency-critical path (real-time voice, sub-100ms response requirements)
  • You’re processing sensitive data and can’t route through an external gateway

References