TL;DR:

  • Exponential backoff with jitter on LLM API calls handles rate limits and transient errors without thundering herds
  • Circuit breakers prevent cascading failures when downstream tools or APIs are degraded
  • Every agent loop needs a maximum iteration budget — a hard exit condition independent of model judgment

Most guidance on building AI agents focuses on the happy path: tools are called correctly, the model produces valid outputs, tasks complete. Production is different. The model times out. The tool returns an error. The JSON is malformed. The agent loops. Understanding the failure modes — and building recovery into the architecture — is what separates agents that run reliably for months from ones that require constant babysitting.

The Failure Taxonomy

Before building recovery mechanisms, it helps to categorise the failures you’re actually defending against:

Transient API failures. LLM provider APIs have rate limits, occasional 500 errors, and connection timeouts. These are expected and temporary — the right response is retry with backoff, not failure propagation.

Malformed model outputs. The model produces a tool call with missing fields, invalid JSON, or parameters that don’t match your schema. Retrying immediately with the same prompt often produces the same malformed output.

Tool failures. The external tool or API the agent calls returns an error, rate limit, or unexpected response format. The agent needs to decide whether to retry, skip, or abort.

Agent loops. The agent repeatedly calls the same tool, receives the same output, and keeps trying without making progress. Without explicit loop detection or iteration limits, these run until you hit a token or cost ceiling.

Context window overflow. On long-running tasks, conversation history grows until it exceeds the model’s context window, causing an error or degraded performance. Agents need context management strategy to prevent this.

Retry Patterns for LLM API Calls

Exponential backoff with jitter is the standard approach for transient LLM API failures:

import asyncio
import random

async def call_llm_with_retry(client, messages, max_retries=3):
    for attempt in range(max_retries):
        try:
            return await client.messages.create(
                model="claude-opus-4-7",
                max_tokens=4096,
                messages=messages,
            )
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            base_delay = 2 ** attempt
            jitter = random.uniform(0, base_delay * 0.1)
            await asyncio.sleep(base_delay + jitter)
        except (APIConnectionError, APITimeoutError):
            if attempt == max_retries - 1:
                raise
            await asyncio.sleep(2 ** attempt)

The jitter prevents thundering herd: if multiple agent instances all hit a rate limit simultaneously and retry at the same interval, they hit the limit again together. Adding randomness spreads the retry attempts.

For malformed model outputs specifically, retry logic needs to modify the prompt rather than repeat it verbatim. Pass the validation error back to the model and ask it to correct the output:

async def get_structured_output(client, prompt, schema, max_retries=2):
    messages = [{"role": "user", "content": prompt}]

    for attempt in range(max_retries):
        response = await client.messages.create(...)

        try:
            return parse_and_validate(response.content, schema)
        except ValidationError as e:
            if attempt == max_retries - 1:
                raise
            messages.append({"role": "assistant", "content": response.content})
            messages.append({
                "role": "user",
                "content": f"Your response had a validation error: {e}. Please correct it."
            })

Circuit Breakers for Tool Calls

A circuit breaker prevents an agent from repeatedly calling a tool that’s known to be failing. The pattern has three states: closed (normal operation), open (tool is failing, calls are rejected immediately), and half-open (testing whether the tool has recovered):

class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=60):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.state = "closed"
        self.last_failure_time = None

    async def call(self, func, *args, **kwargs):
        if self.state == "open":
            elapsed = time.time() - self.last_failure_time
            if elapsed > self.recovery_timeout:
                self.state = "half-open"
            else:
                raise CircuitOpenError(
                    f"Tool unavailable, retry in {self.recovery_timeout - elapsed:.0f}s"
                )

        try:
            result = await func(*args, **kwargs)
            if self.state == "half-open":
                self.state = "closed"
                self.failure_count = 0
            return result
        except Exception:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.failure_threshold:
                self.state = "open"
            raise

When a circuit is open, the agent should either skip the tool and proceed with what it has, queue the task for retry, or escalate to a human. The right choice depends on whether the tool is essential to the task.

Hard Exit Conditions

The most important safety mechanism in any agent loop is a hard maximum iteration count that the model cannot override:

MAX_ITERATIONS = 25
MAX_TOOL_CALLS = 50
MAX_DURATION_SECONDS = 300

async def run_agent_loop(agent, task):
    start_time = time.time()
    iteration = 0
    total_tool_calls = 0

    while True:
        iteration += 1

        if iteration > MAX_ITERATIONS:
            return AgentResult(status="max_iterations", partial_output=agent.accumulated_output)

        if total_tool_calls > MAX_TOOL_CALLS:
            return AgentResult(status="max_tool_calls", partial_output=agent.accumulated_output)

        if time.time() - start_time > MAX_DURATION_SECONDS:
            return AgentResult(status="timeout", partial_output=agent.accumulated_output)

        response = await agent.step()
        total_tool_calls += response.tool_calls_made

        if response.is_complete:
            return AgentResult(status="success", output=response.final_output)

These limits should be task-specific where possible. A research task might legitimately need 30 tool calls. A code formatting task making 30 tool calls is almost certainly looping. The limits protect against worst-case behaviour without crippling legitimate workloads.

Graceful Degradation

Not all failures should be errors. An agent that can partially complete a task and return useful partial output is more valuable than one that aborts entirely at the first problem. Build your agents to distinguish between:

Hard failures — the task cannot proceed without this resource (missing authentication credentials, required data not accessible). Return an error immediately.

Soft failures — a component failed but the task can continue with degraded output (one data source unavailable, one enrichment step failed). Record the failure, skip the step, continue.

Ambiguous failures — the right response depends on business logic (the tool returned data but it looks wrong). Log it, continue with a flag, and let a downstream process or human review.

The practical implementation is a structured result type:

@dataclass
class AgentResult:
    status: Literal["success", "partial", "failed"]
    output: Any
    warnings: list[str] = field(default_factory=list)
    errors: list[str] = field(default_factory=list)
    skipped_steps: list[str] = field(default_factory=list)

Loop Detection

Beyond hard iteration limits, detecting when an agent is looping without progress requires tracking what it has actually done. A simple approach: hash each tool call and abort if the same hash appears more than N times:

from collections import Counter
import hashlib, json

class LoopDetector:
    def __init__(self, max_repeated_calls=3):
        self.call_counts = Counter()
        self.max_repeated_calls = max_repeated_calls

    def check(self, tool_name, arguments):
        call_hash = hashlib.md5(
            json.dumps({"tool": tool_name, "args": arguments}, sort_keys=True).encode()
        ).hexdigest()
        self.call_counts[call_hash] += 1
        if self.call_counts[call_hash] > self.max_repeated_calls:
            raise LoopDetectedError(
                f"Tool {tool_name} called with identical arguments {self.max_repeated_calls}+ times"
            )

This catches the most common loop pattern — the agent retrying the same failing call — without flagging legitimate repeated calls to different endpoints or with different parameters.

Production agent reliability is mostly about preparation for the failure modes you know are coming. The patterns here aren’t exotic — they’re standard distributed systems reliability patterns applied to the specific characteristics of LLM-based agents. The details that matter: matching retry logic to failure type, using circuit breakers to prevent cascading failures, and ensuring hard exits exist independently of model judgment.