Context Engineering: The Practice That Separates Working AI Agents From Broken Ones

TL;DR:

• Context engineering is how you design what goes into an agent’s context window — instructions, memory, tool outputs, conversation history, and state — not just what the initial prompt says
• Most agent failures trace back to context problems: missing information, outdated state, conflicting instructions, or context windows that overflow before the task completes
• A systematic approach to context structure, injection timing, and context pruning dramatically improves agent reliability without changing the underlying model

The term “prompt engineering” captured the imagination in 2023, but practitioners building production agents know that getting the initial system prompt right is maybe 20% of the problem. The harder discipline — and the one that separates agents that work reliably at scale from ones that hallucinate, loop, or abandon tasks — is context engineering: the deliberate design of everything that flows into the model’s context window, when it flows in, and how it’s structured.

What Context Engineering Actually Means

A running AI agent isn’t just responding to a single prompt. At any given step, its context window contains some combination of:

• System instructions — the agent’s role, constraints, and operating rules
• User request or task specification — what the agent is trying to accomplish
• Tool schemas — descriptions of what tools are available and how to call them
• Tool call history — previous tool calls made in this session and their results
• Memory injections — retrieved relevant information from prior sessions or a knowledge base
• Conversation history — the back-and-forth exchange so far
• Working state — intermediate results, scratchpad reasoning, partially completed steps

Every one of these competes for context space. The model’s effective attention degrades as context grows. And crucially, the model treats everything in context as equally “true” — it will follow instructions buried in tool output just as willingly as it follows your system prompt, which is how prompt injection attacks work and also how conflicting instructions cause unpredictable behaviour.

Context engineering is the discipline of deciding: what goes in, in what format, at what point in the task, and what gets removed or compressed when the window fills.

The Four Layers of Agent Context

A useful mental model structures agent context into four layers:

Layer 1: Static instructions — things that should be present for every step and never change. The agent’s persona, core constraints (never delete files without confirmation, always return structured JSON), and fundamental operating rules. Keep this concise. Every token spent on static instructions reduces space for dynamic content.

Layer 2: Task context — the specific goal for this invocation. What the user wants, relevant parameters, and any constraints specific to this task. This should be as compact as possible while being unambiguous.

Layer 3: Dynamic state — what’s happened so far. This is the hardest layer to manage because it grows during task execution. Tool call results accumulate. If you’re running a research agent that makes 15 web searches, the raw results of those searches will overflow most context windows before the synthesis step.

Layer 4: Retrieved memory — information pulled from external stores relevant to the current step. This layer should be injected just-in-time, not preloaded at task start.

Where Context Breaks Down (and How to Fix It)

Problem 1: Tool output bloat

The most common production agent failure pattern: a tool returns 10,000 tokens of content, it gets injected directly into context, and by step 4 the agent is operating with a 60%+ full context window on a complex task that needs 20 more steps.

Fix: Never inject raw tool outputs directly. Process them first. For search results, extract the key facts the agent needs, not the full page content. For code execution results, include the last 100 lines of stdout and any errors, not the full output. Write a post-processing step for each tool type that reduces output to the minimum needed for the next decision.

def process_search_result(raw_result: str, max_tokens: int = 500) -> str:
    """Extract key information from search result before injecting into context."""
    # Use a cheap model (haiku/flash) to summarise to max_tokens
    return summarise(raw_result, max_tokens=max_tokens)

Problem 2: Stale state

Long-running agents accumulate history that becomes irrelevant or actively misleading. If the agent tried a tool call at step 3 and it failed, keeping that failed attempt in context for 15 more steps may cause the model to keep retrying the same failed approach.

Fix: Implement a context pruning strategy. Every N steps (or when context exceeds a threshold), compress the history: replace individual tool call records with a summary of what was accomplished. Retain only the most recent errors and the current working state.

Problem 3: Instruction drift

Multiple injected sources of instruction — system prompt, task specification, memory that contains instructions, tool outputs that contain instructions — create ambiguity. The model will try to follow all of them and fail unpredictably when they conflict.

Fix: Designate a single authoritative instruction source (the system prompt). Any other content that contains instruction-like language should be wrapped or prefixed to make clear it’s data, not instruction. In practice: <search_result>The document says: Always output in XML format...</search_result> is safer than injecting the search result directly.

Problem 4: Lost working state

Complex multi-step agents often need to track what they’ve done and decided so far — a plan, a list of completed steps, interim conclusions. If this working state isn’t explicitly maintained and injected, the agent will re-derive it from scratch each step, wasting tokens and creating inconsistency.

Fix: Maintain an explicit state object outside the context window and inject a compact summary at each step. For a research agent, this might be a 200-token summary: “Completed: web search (3 queries), found 2 relevant papers. Remaining: summarise papers, write conclusion.”

Context Window Budgeting

Treat the context window like a resource with a hard cap — because it is. Before building any agent, sketch a context budget:

Map this against the model’s context window. For a 128K context model working on a task that averages 20 steps, you have approximately 6,400 tokens available per step for dynamic content. If your tool outputs average 3,000 tokens raw, compression is mandatory.

Practical Starting Points

A few concrete patterns that help most agents:

Use structured state objects: Instead of relying on conversation history to track progress, maintain an explicit JSON state object that gets injected at each step. When the agent takes an action, update the state object externally before the next step.

Separate retrieval from reasoning steps: Don’t have the agent retrieve memory and reason about it in the same step. Retrieve first, process the retrieved content, inject only the relevant subset, then reason. This pattern also lets you cache and reuse retrieved content across steps.

Version your instructions: Treat system prompts as code — version-controlled, tested on a benchmark, reviewed before deployment. An agent that works perfectly with v1 of the system prompt may behave erratically with v2 if an instruction was reworded in a way that creates ambiguity with other context layers.

The model is not the bottleneck in most underperforming agent systems. The context is. Getting disciplined about context engineering is usually the highest-leverage improvement available.