TL;DR:
- AI agents degrade measurably after around 35 minutes of tool calls and context accumulation — a phenomenon sometimes called “context rot.”
- Three strategies dominate production deployments: periodic summarisation, retrieval-augmented memory (retrieving only what’s relevant per step), and prompt caching to reduce cost on repeated context.
- The best architecture depends on task structure: linear tasks suit summarisation; branching or parallel tasks suit retrieval-based memory.
Anyone building long-running AI agents hits the same wall eventually. The agent works brilliantly for the first twenty minutes, then starts repeating itself, forgetting tool output from earlier in the session, or making decisions that contradict things it said ten steps back. The model hasn’t changed. The context has.
Context window management is one of the least glamorous parts of agent engineering and one of the most consequential for production reliability.
What “Context Rot” Actually Means
Large language models are stateless. They receive a context window, generate a response, and the session state is entirely contained in the tokens passed in. When an agent runs tool calls, each result gets appended to the context. After enough steps, the context is dominated by tool outputs, intermediate reasoning, and scaffolding — and the model’s effective attention to the actual task objective degrades.
Research from several teams has identified roughly 35 minutes or 50–80 tool calls as the point where single-context agents show measurable performance drops on multi-step benchmarks. Beyond that threshold, error rates increase, agents start hallucinating prior tool outputs, and coherence with the original task objective weakens.
The underlying cause is attention dilution: with a long context, the model can technically “see” the original instruction, but its effective weighting of that instruction is diminished relative to the volume of intermediate content.
Strategy 1: Periodic Summarisation
The simplest fix. At a defined interval — every N steps or when context exceeds a threshold — the agent pauses and summarises its progress to date, then replaces the detailed tool call history with the summary.
[System prompt]
[Original task]
[Summary of completed steps as of turn N: <summary>]
[Last K tool calls and results: <recent history>]The summary needs to preserve:
- What the original objective was
- What has been completed and what the results were
- What remains to be done
- Any constraints or decisions made along the way
Done well, summarisation extends agent effective context by 3–5× with minimal performance cost. Done badly — if the summary loses critical details from earlier tool outputs — it introduces errors that compound.
Best for: Linear, sequential tasks where earlier steps don’t need to be re-examined once summarised.
Strategy 2: Retrieval-Augmented Memory
Instead of keeping everything in-context, store tool outputs, observations, and intermediate results in a vector store. At each step, retrieve the K most relevant items from memory before generating the next action.
[System prompt]
[Original task]
[Retrieved relevant context: <top-K relevant items>]
[Current step input]This keeps the active context small and task-focused. The agent “remembers” everything in theory, but only loads what’s relevant to the current step. The quality of retrieval determines the quality of the agent’s effective memory.
Best for: Tasks with many parallel threads or where the agent needs to revisit decisions from much earlier in the session. Research agents, long-form content generation, and multi-document analysis all benefit.
Limitation: Retrieval adds latency and introduces retrieval errors — the agent can miss relevant context if the embedding distance doesn’t reflect semantic relevance to the current step.
Strategy 3: Hierarchical Memory
A refinement of retrieval-based approaches. Rather than a flat vector store, maintain three tiers of memory:
- Working memory: the active context window, containing the current step and recent tool calls
- Episodic memory: a summary buffer of completed sub-tasks, rotated in as needed
- Semantic memory: a persistent vector store of facts, decisions, and reference material extracted from the session
The agent routes queries to the appropriate memory tier based on the type of retrieval. Facts (“what was the API response from step 12?”) go to episodic memory; general knowledge (“what format does this API expect?”) might go to semantic memory; active reasoning stays in working memory.
This architecture is more complex to implement but achieves better performance on long-horizon tasks than either pure summarisation or flat retrieval.
Strategy 4: Prompt Caching
Not a memory management strategy in the strict sense, but essential for cost and latency when you have a large static context (a long system prompt, reference documents, or base instructions that don’t change step-to-step).
Anthropic’s API, OpenAI’s API, and Google’s Gemini all support prompt caching — marking a prefix of the context as cacheable, so repeated calls with the same prefix don’t re-process those tokens. For agents with a large static system prompt plus dynamic task content, this can reduce per-call cost by 60–80% on the cached portion.
The practical pattern: keep your static context (instructions, reference material, examples) in the cacheable prefix, and append dynamic content (tool results, current task state) after the cache boundary.
Choosing Your Approach
| Task type | Recommended strategy |
|---|---|
| Short sequential tasks (<30 steps) | No management needed |
| Long sequential tasks (>50 steps) | Periodic summarisation |
| Parallel/branching tasks | Retrieval-augmented memory |
| Long-horizon research tasks | Hierarchical memory |
| Any task with large static context | Prompt caching (combine with above) |
What Most Teams Get Wrong
The most common mistake is treating context management as a problem to solve later, once an agent is “working.” By the time you retrofit context management into a production agent, you’re often reworking the entire scaffolding.
Design for context from the start. Decide your context boundary before you write the agent loop. If you’re building a workflow that will run 50+ tool calls, your architecture needs to account for that from the initial design — not as a patch once you’ve hit the 35-minute wall in production.
The second mistake is over-engineering. Most production agents doing linear tasks don’t need hierarchical memory — they need clean summarisation at a sensible interval. Match the strategy to the task complexity, not to the most sophisticated option available.