TL;DR:

  • Token spend in multi-agent systems compounds fast — a single runaway loop can exhaust a monthly budget in hours
  • Implement cost caps at the agent, task, and user levels independently; one cap isn’t enough
  • LLM gateway proxies (LiteLLM, Portkey, OpenRouter) give you rate limiting, fallbacks, and spend tracking in one layer

Running an AI agent in a demo is cheap. Running one in production against real workloads, with real users, on a real LLM provider API is something else. The compounding nature of multi-step agent loops — each tool call triggering another LLM call, which triggers another tool call — means that a bug, an unexpected input, or a malicious prompt can burn through your API budget at a rate no human could manually replicate.

This guide covers the practical patterns for keeping costs under control without crippling agent capability.

Why Agent Cost Problems Are Different

Single-turn LLM apps have a predictable cost profile: one prompt in, one completion out, some multiplier for the number of users. You can model this.

Agents don’t work that way. An agent completing a task might make anywhere from 3 to 300 LLM calls depending on the complexity of the task, the quality of tool definitions, and whether anything goes wrong mid-run. A tool that silently returns an error will cause a well-designed agent to retry. An ambiguous goal will cause it to clarify, then plan, then re-plan. Without hard limits, that loop continues.

The failure mode isn’t usually malicious — it’s that you didn’t anticipate a combination of inputs, the agent interprets something broadly, and a task that should have cost $0.08 ends up costing $12. Across enough concurrent users, that’s a bill problem, not a rounding error.

Layer 1: Per-Call Token Limits

The first control is the most basic: cap the tokens on every LLM call. Set max_tokens explicitly on every completion request — never let it default. For agent reasoning calls, 1,000–2,000 tokens is usually sufficient. For synthesis or summarisation steps, you may need more, but set it deliberately.

Also set a maximum context window budget per agent run. If your agent is accumulating conversation history, implement a context trimming strategy (sliding window, summarisation, or selective eviction) so the input tokens per call don’t grow indefinitely.

Layer 2: Per-Task Budget Caps

Every agent task should have a maximum spend limit set before it starts running. Implement this as a token counter that increments with every API call made during the task. When the counter exceeds the budget, the agent should either:

  • Return a partial result with a clear explanation of why it stopped
  • Escalate to a human for input
  • Fail gracefully with a budget-exceeded error that the calling system can handle

In LangGraph, this integrates naturally into the state object — maintain a tokens_used field and check it at each node. In CrewAI and AutoGen, implement it as a callback or middleware layer that wraps the LLM client.

The budget should vary by task type. A quick lookup task gets a tight budget. A deep research task gets more. Set these explicitly in your task definitions, not as a single global limit.

Layer 3: User and Tenant Quotas

If your agent product has multiple users or tenants, you need quota enforcement at the user level as well as the task level. This prevents a single heavy user from exhausting shared API capacity and degrading the experience for everyone else.

Implement this with a token ledger per user, persisted in Redis or a database. Before starting any agent run, check the user’s remaining quota. After completion, debit the actual token usage. Give users visibility into their consumption via a dashboard or API endpoint — the support burden of unexpected quota exhaustion is significant if users can’t see it coming.

Layer 4: LLM Gateway for Cross-Cutting Controls

For teams running multiple agents across multiple models, a LLM gateway proxy handles rate limiting, cost tracking, and fallback routing at the infrastructure level rather than in each agent implementation.

LiteLLM is the most widely deployed open-source option. It supports per-key rate limits, spend budgets, model fallbacks, and a dashboard for spend tracking. You route all LLM calls through the proxy, and it enforces the limits centrally.

Portkey and OpenRouter offer similar functionality as managed services, with additional features like semantic caching (which can cut costs significantly for agents with repetitive sub-tasks) and automatic retry with exponential backoff.

The proxy approach is worth the added infrastructure complexity for anything beyond a single agent in production. It gives you one place to adjust limits, add new models, and monitor spend across the entire system.

Layer 5: Semantic Caching

Many agent workloads involve repeated or near-identical LLM calls — the same tool result interpretation, the same planning step for similar tasks, or the same RAG retrieval for overlapping queries. Semantic caching stores LLM responses and returns the cached result for semantically similar future queries, bypassing the API call entirely.

Redis with a vector index (using RedisVL or similar) is a common implementation. Caching hit rates vary by workload, but for applications with overlapping queries — a customer support agent, a document Q&A system, a code review agent — hit rates of 20–40% are achievable. At that rate, semantic caching is one of the highest-ROI cost reduction techniques available.

Monitoring and Alerting

Put spend alerting in place before you need it. Set alerts at 50%, 75%, and 90% of your monthly budget. Alert on unexpected spikes: if agent spend in a 10-minute window is 10x the historical baseline, something is wrong.

Track cost per task completion, not just cost per token. This gives you a metric that ties infrastructure spend to business output and makes it obvious when a workflow change increases cost without a corresponding increase in value.

Practical Defaults to Start With

If you’re starting from scratch, these are reasonable defaults:

  • Max tokens per LLM call: 2,000 for reasoning, 4,000 for synthesis
  • Max tokens per task: 50,000 (roughly $0.15–0.50 depending on model)
  • Per-user daily quota: size to your pricing model, then halve it
  • Alert threshold: 80% of monthly budget
  • Semantic cache TTL: 24 hours for stable content, 1 hour for dynamic queries

Adjust these based on your actual workload profile — but having explicit numbers beats having nothing.