If you’re building AI agents and you haven’t looked at prompt caching yet, there’s a good chance you’re leaving a substantial amount of money on the table. For agents with long system prompts, tool definitions, or context that doesn’t change between calls — which is basically every production agent — prompt caching can cut your input token costs by 80-90% on the cached portion. That’s not a marginal improvement. At scale, it changes the economics of running agents completely.
Here’s the thing: prompt caching doesn’t require you to restructure your architecture. It’s an API-level feature that you opt into by marking parts of your prompt as cacheable. The model provider handles the rest.
How Prompt Caching Works
When you send a prompt to an LLM API without caching, every token in that prompt is processed from scratch on every request. For agents, that typically means your system prompt (often 500-2000 tokens), your full tool schema (another 500-3000 tokens depending on how many tools you have), and any persistent context you’re passing gets charged and processed on every single call.
Prompt caching changes this. You mark specific blocks of your prompt as cache-eligible. On the first call, those blocks get processed and cached on the provider’s infrastructure. On subsequent calls within the cache lifetime, those blocks are retrieved from cache rather than reprocessed. You pay a fraction of the normal input token price for the cached portion.
With Anthropic’s implementation, cached tokens cost 10% of the normal input price (a 90% discount). Cache writes cost 25% more than normal tokens, but this only applies to the first call that populates the cache. The cache lasts for five minutes by default, extendable to an hour with explicit configuration. For agents processing multiple turns in a session, this means every call after the first benefits from cached pricing.
OpenAI’s prompt caching works slightly differently — it’s automatic for prompts over 1024 tokens and you don’t need to explicitly mark blocks. The discount is 50% on cached tokens. The cache lifetime is shorter (unclear, but typically within a session).
Where the Savings Come From
The biggest savings are on the parts of your prompt that are identical across calls:
System prompts: Your agent’s instructions, persona, and constraints. These are fixed across all calls. For a Claude agent with a 1500-token system prompt running 1000 calls per day, caching that prompt saves roughly 1350 tokens per call — 1.35 million tokens per day — at a 90% discount. At claude-opus-4-7 pricing, that’s a meaningful daily saving.
Tool definitions: Every tool you give your agent gets included in the prompt as a JSON schema. A typical agent with ten tools might have 2000 tokens of tool definitions. These never change mid-session. Cache them.
Retrieved context that’s reused across calls: If you’re fetching a large document or knowledge base at the start of a session and referencing it across multiple turns, that’s a caching opportunity. You fetch it once and mark it as cacheable; subsequent calls within the cache window pay the reduced rate.
Implementation with Anthropic’s API
Anthropic’s caching uses cache_control blocks. You add "cache_control": {"type": "ephemeral"} to the content block you want cached:
import anthropic
client = anthropic.Anthropic()
# System prompt with caching
system = [
{
"type": "text",
"text": """You are a customer support agent for Acme Corp.
Your job is to help customers with their orders, returns, and account questions.
Always be polite and direct. If you cannot resolve an issue, escalate to a human agent.
Company policies:
[... 1200 tokens of policy content ...]
""",
"cache_control": {"type": "ephemeral"}
}
]
# Tool schema with caching — cache the last tool to cache everything above it
tools = [
{
"name": "lookup_order",
"description": "Look up a customer order by order ID",
"input_schema": {
"type": "object",
"properties": {
"order_id": {"type": "string", "description": "The order ID"}
},
"required": ["order_id"]
}
},
{
"name": "process_refund",
"description": "Process a refund for an order",
"input_schema": {
"type": "object",
"properties": {
"order_id": {"type": "string"},
"reason": {"type": "string"}
},
"required": ["order_id", "reason"]
},
"cache_control": {"type": "ephemeral"} # Cache everything up to and including this tool
}
]
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
system=system,
tools=tools,
messages=[{"role": "user", "content": "I need to return order #12345"}]
)
# Check cache usage in the response
usage = response.usage
print(f"Input tokens: {usage.input_tokens}")
print(f"Cache read tokens: {usage.cache_read_input_tokens}")
print(f"Cache write tokens: {usage.cache_creation_input_tokens}")One important detail: Anthropic’s cache control marks everything up to and including that block as cacheable. So if you put cache_control on your last tool, the system prompt and all tools above it are included in the cache. The cache hit requires an exact match on all cached content — including token count and model version.
Multi-Turn Agents and Conversation History
For multi-turn agents, the strategy shifts slightly. Your system prompt and tools stay cached across the whole session. The conversation history grows with each turn, which means the cacheable portion stays the same but the non-cached portion grows.
A common pattern: mark the conversation history up to (but not including) the last two turns as cacheable. This way you get caching on all the historical context that won’t change, while paying full price only for the most recent exchange.
def build_messages_with_caching(history: list[dict], new_message: str) -> list[dict]:
messages = []
# Cache all but the last two turns of history
for i, msg in enumerate(history):
if i < len(history) - 2:
# Add cache control to the last message in the cacheable range
if i == len(history) - 3:
msg_with_cache = {**msg, "cache_control": {"type": "ephemeral"}}
messages.append(msg_with_cache)
else:
messages.append(msg)
else:
messages.append(msg)
messages.append({"role": "user", "content": new_message})
return messagesWhen Not to Bother
Prompt caching is worth the implementation effort when your cached tokens are large relative to your dynamic tokens, and when you’re making repeated calls within the cache window. It’s less valuable when:
- Your system prompts are short (under 200 tokens)
- You’re making calls very infrequently — a single call every few minutes will often miss the cache window entirely
- Your tool schemas change dynamically (though this is unusual in practice)
It’s also worth knowing that cache misses still charge you the write cost. If your cache hit rate is low because calls are spread across long time windows, the write overhead can outweigh the savings. Monitor cache_read_input_tokens vs cache_creation_input_tokens in your usage logs to check your effective hit rate.
Practical Impact
A real-world example: a document analysis agent with a 2000-token system prompt, 1500-token tool schema, and 3000 tokens of shared document context (6500 tokens total cached) running 10,000 calls per day saves around 58.5 million tokens per day at the 90% discount rate — a very significant reduction in the daily bill. Even modest agents with 1000 daily calls see savings that make implementation time worthwhile within weeks.
Prompt caching is one of those optimisations that’s easy to dismiss because it sounds like a minor API detail. In practice, for any production agent running at meaningful volume, it’s one of the highest-leverage cost improvements available.