Most agent pipelines start simple: one model, one provider, OpenAI or Anthropic, and you call it directly. Then production happens. You need a fallback model for when rate limits hit. You want to run cheaper models for low-priority tasks and expensive ones for complex reasoning. Your team wants visibility into what each agent call is costing. You want to A/B test providers without changing application code.

At this point, every team reinvents the same infrastructure. LiteLLM Proxy means you don’t have to.

What LiteLLM Proxy Does

LiteLLM is a Python library that provides a unified interface to 100+ LLM providers — OpenAI, Anthropic, Google Gemini, Cohere, Azure OpenAI, Bedrock, Ollama local models, and many more — behind an API that matches OpenAI’s format. Your agent code calls what looks like an OpenAI endpoint, and LiteLLM handles routing it to whichever underlying model you’ve configured.

The proxy mode takes this further. Rather than importing LiteLLM as a library in each service, you run it as a standalone HTTP server. Your agents and other services point their base_url at the proxy, and it becomes a centralised gateway for all LLM traffic in your stack.

# litellm_config.yaml
model_list:
  - model_name: fast
    litellm_params:
      model: gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

  - model_name: fast
    litellm_params:
      model: claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: reasoning
    litellm_params:
      model: claude-opus-4-7
      api_key: os.environ/ANTHROPIC_API_KEY
litellm --config litellm_config.yaml --port 4000

Your agent then calls:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000", api_key="anything")
response = client.chat.completions.create(
    model="fast",
    messages=[{"role": "user", "content": "Summarise this document..."}]
)

The agent code has no knowledge of which underlying provider handled the request. Swap providers, add fallbacks, change routing rules — all in the proxy config, without touching the agents.

Load Balancing and Fallbacks

The most immediately useful feature for production agents is the ability to list multiple deployments for the same logical model name. LiteLLM will load balance across them and fall back automatically on errors.

model_list:
  - model_name: fast
    litellm_params:
      model: gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
    model_info:
      rpm_limit: 500

  - model_name: fast
    litellm_params:
      model: claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
    model_info:
      rpm_limit: 500

When your agent calls model: "fast", LiteLLM distributes traffic across the two providers by least-busy routing. If OpenAI returns a rate limit error, the proxy automatically retries against Anthropic. From the agent’s perspective, the call just succeeds.

This matters more than it might sound in multi-agent systems where dozens of parallel agent tasks are all making LLM calls simultaneously. A single provider rate limit can cascade into failed tasks; a load-balanced gateway handles it transparently.

Per-Request Budget Enforcement

LiteLLM Proxy supports spend tracking and budget limits at the API key level. You create virtual keys through the proxy, each with its own spending budget.

# Create a key with a $10 daily limit
curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer your-master-key" \
  -d '{"max_budget": 10, "budget_duration": "1d"}'

Different agents or teams get different keys, each capped independently. An out-of-control agent that loops and makes thousands of calls won’t drain your entire organisation’s API budget — it’ll hit its own limit and fail cleanly. The proxy logs every call with token counts and cost estimates, so you get per-agent cost attribution without instrumenting each agent individually.

Observability Without Code Changes

The proxy has built-in integrations with observability platforms: Langfuse, LangSmith, Helicone, and others. You configure the integration once in the proxy config and every LLM call across your entire agent stack is traced automatically.

general_settings:
  master_key: "sk-your-master-key"

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

environment_variables:
  LANGFUSE_PUBLIC_KEY: "pk-..."
  LANGFUSE_SECRET_KEY: "sk-..."

The traces include model, tokens in/out, latency, cost, and the full prompt/completion if you want it. No code changes required in the agents themselves — the proxy instruments everything at the gateway layer.

When to Use the Proxy vs the Library

The LiteLLM Python library (imported directly, not the proxy server) is useful when you’re building a single agent and just want to avoid writing provider-specific code. But once you have multiple agents, multiple services, or a team working on the same stack, the proxy makes more sense:

  • Centralised configuration: change models and routing in one place
  • Cross-service cost tracking: all agents share one observability view
  • Gateway-level guardrails: rate limiting, budget caps, and content filtering applied uniformly
  • Independent deployability: the proxy runs as its own service; agents don’t need updates to benefit from gateway improvements

The proxy adds a network hop (typically adding 5-20ms of latency), which is negligible against the 500ms-10s latency of LLM calls themselves. For most production agent pipelines, the operational benefits far outweigh the overhead.

Docker Deployment

For production, run the proxy as a container:

docker run -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main \
  --config /app/config.yaml

Add a PostgreSQL connection for persistent spend tracking across restarts. The proxy’s dashboard (available at /ui on the proxy host) gives you a real-time view of spend by key, model usage, and error rates.

LiteLLM Proxy doesn’t solve every problem in agent infrastructure — it’s a gateway, not an orchestrator — but it addresses a cluster of recurring production issues (provider fallback, cost visibility, multi-model routing) in a single, low-configuration service. For teams running more than two or three agents in production, it’s worth the setup time.