Semantic Caching for AI Agents: Cutting LLM Costs Without Sacrificing Quality

TL;DR:

• Exact-match caching is nearly useless for LLM agents — “What’s the weather?” and “What is the current weather?” are different strings but mean the same thing
• Semantic caching embeds the incoming query and checks vector similarity against cached responses, serving the cache hit when similarity exceeds a threshold
• Libraries like GPTCache, Redis with vector support, and LangChain’s built-in cache make this straightforward to add to existing agent pipelines

LLM API costs are the number one operational concern for teams running AI agents in production. A single agent task might invoke a model dozens of times. At scale, those costs compound fast. The instinct is to cache — but naive key-value caching doesn’t work well for natural language, because users rarely phrase identical queries.

Semantic caching solves this. Instead of comparing query strings exactly, you compare their meaning. Two queries that ask the same thing but phrase it differently share a high cosine similarity in embedding space. If they’re similar enough, you serve the cached response from the first query rather than paying for a new model call.

In practice, teams implementing semantic caching in agent pipelines see 40–70% reduction in LLM API calls depending on the nature of the workload. Read-heavy tasks — FAQ agents, document Q&A, classification pipelines — see the highest cache hit rates. Write-heavy tasks — code generation, creative content — see lower rates but still benefit for repeated subtasks.

How Semantic Caching Works

The basic flow:

1. An incoming query arrives at your agent
2. You generate an embedding of the query using a fast, cheap embedding model (text-embedding-3-small or similar)
3. You search your vector cache for stored embeddings with cosine similarity above your threshold (typically 0.90–0.95)
4. If a match is found: return the cached response immediately, no LLM call
5. If no match: call the LLM, store the (embedding, response) pair in the cache

The threshold is the key tuning parameter. Too low (0.80) and you’ll serve semantically related but subtly different queries with the same response, which can introduce errors. Too high (0.99) and you’re back to near-exact matching. Start at 0.92 and adjust based on your domain’s semantic variance.

GPTCache: The Dedicated Library

GPTCache from Zilliz is the most mature semantic caching library for LLM applications. It supports multiple embedding backends, similarity engines, and cache storage backends.

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.similarity_evaluation import SearchDistanceEvaluation
from gptcache.manager import CacheBase, VectorBase, get_data_manager

# Initialize with ONNX embedding (fast, local) and Redis vector backend
onnx = Onnx()
data_manager = get_data_manager(
    CacheBase("sqlite"),         # metadata store
    VectorBase("redis",          # vector store
        host="localhost",
        port=6379,
        password="",
        dimension=onnx.dimension
    )
)

cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(),
    similarity_threshold=0.92,
)

# Drop-in replacement for OpenAI calls
response = openai.ChatCompletion.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is the capital of France?"}]
)

On the second call asking “What’s France’s capital?” or “Tell me the capital city of France”, GPTCache intercepts and returns the cached response before the API call is made.

Redis Vector Search as a Cache Backend

Redis with the Redis Stack or Redis Enterprise’s vector similarity module is a good production choice — low latency, familiar operations tooling, and TTL support built in. You can manage the semantic cache yourself without a dedicated library:

import redis
import numpy as np
from openai import OpenAI

r = redis.Redis(host="localhost", port=6379, decode_responses=False)
client = OpenAI()

def get_embedding(text: str) -> list[float]:
    return client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    ).data[0].embedding

def semantic_cache_lookup(query: str, threshold: float = 0.92) -> str | None:
    query_embedding = get_embedding(query)
    
    # Search for similar cached queries
    results = r.execute_command(
        "FT.SEARCH", "cache-idx",
        f"*=>[KNN 1 @embedding $vec AS score]",
        "PARAMS", "2", "vec", np.array(query_embedding, dtype=np.float32).tobytes(),
        "SORTBY", "score",
        "RETURN", "2", "response", "score"
    )
    
    if results and len(results) > 1:
        doc = results[1]
        score = float(doc[doc.index(b"score") + 1])
        if score >= threshold:
            return doc[doc.index(b"response") + 1].decode()
    return None

def cached_llm_call(query: str, **kwargs) -> str:
    # Check cache first
    cached = semantic_cache_lookup(query)
    if cached:
        return cached
    
    # Call LLM
    response = client.chat.completions.create(
        messages=[{"role": "user", "content": query}],
        **kwargs
    ).choices[0].message.content
    
    # Store in cache with TTL
    embedding = get_embedding(query)
    r.hset(f"cache:{hash(query)}", mapping={
        "query": query,
        "response": response,
        "embedding": np.array(embedding, dtype=np.float32).tobytes()
    })
    r.expire(f"cache:{hash(query)}", 86400)  # 24h TTL
    
    return response

LangChain’s Built-In Semantic Cache

If you’re already using LangChain, semantic caching is a one-liner:

from langchain.globals import set_llm_cache
from langchain_community.cache import RedisSemanticCache
from langchain_openai import OpenAIEmbeddings

set_llm_cache(RedisSemanticCache(
    redis_url="redis://localhost:6379",
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
    score_threshold=0.92,
))

# All subsequent LLM calls through LangChain check the semantic cache automatically

This works transparently across any LangChain-compatible LLM. Every call goes through the cache layer before hitting the provider API.

What to Cache and What Not To

Semantic caching works best for queries where you expect semantic repetition. These are good candidates:

• Customer support agents: Users ask variants of the same questions repeatedly
• Document Q&A: “Summarise section 3” vs “Give me a summary of the third section”
• Classification pipelines: Sentiment, intent, category classification on similar inputs
• Knowledge base retrieval: Lookup-style queries with stable answers
• Translation and language tasks: Similar source texts produce similar translations

These are poor candidates:

• Code generation with context: The query might be similar but the surrounding codebase context changes the correct answer
• Real-time data queries: Anything requiring current information (weather, prices, stock data) should never be cached beyond seconds
• Creative generation: Deliberately want variance — caching defeats the purpose
• Personalized responses: When the correct response varies by user state or history

TTL Strategy

Cached responses go stale. Set TTLs based on the expected stability of the answer:

Always surface cache staleness to the agent’s reasoning layer when it matters — an agent deciding whether to answer “who leads company X” should know if the cached answer is two weeks old.

Monitoring Cache Performance

Track these metrics in production:

• Hit rate: cache hits / total queries. Target 40%+ for FAQ-style agents
• Latency delta: cached response time vs. live LLM response time (usually 100x faster)
• Cost saved: (cache hits × average cost per call)
• Threshold drift: track when similarity scores cluster — if you’re seeing lots of 0.91 scores being rejected, your threshold may be too high

GPTCache exports hit rate metrics by default. For Redis-based setups, instrument your cache lookup function with your existing observability stack.

The Bottom Line

Semantic caching is one of the highest-ROI optimizations available for production AI agents. The implementation is straightforward, the libraries are mature, and the cost savings are immediate. If your agent handles any volume of user queries with semantic repetition, start with GPTCache or LangChain’s Redis semantic cache and tune from there.