Standard RAG works well for a lot of things. You chunk documents, embed them, store the vectors, and at query time you retrieve the most semantically similar chunks and shove them in the context window. For factual questions that can be answered by reading a single passage — “what’s the refund policy?” or “what does this clause say?” — it’s genuinely useful.

But standard RAG has a quiet failure mode that only shows up when queries get complex. Ask it to summarise themes across an entire document collection, identify which entities appear most frequently across multiple reports, or explain how several concepts relate to each other, and it struggles. The chunk retrieval model finds locally relevant pieces but misses the global structure of the information.

GraphRAG, developed by Microsoft Research and open-sourced in 2024, is an attempt to fix this.

The Core Idea

Standard RAG treats documents as bags of chunks. GraphRAG treats them as a graph of entities and relationships.

During indexing, GraphRAG runs an LLM over the source documents to extract entities (people, organisations, concepts, locations, dates) and the relationships between them. These entities and relationships are stored as a knowledge graph. At the same time, it clusters related entities into communities — groups of nodes that are densely connected — and generates summaries at multiple levels of abstraction.

When a query comes in, GraphRAG can answer in two modes:

Local search finds the specific entities and relationships most relevant to a query, then uses them to ground a response. This is similar to standard RAG but with graph traversal rather than pure vector similarity.

Global search works differently. Instead of retrieving chunks, it queries the community summaries — pre-generated summaries of related entity clusters — and aggregates insights across them. This is what enables questions like “what are the main themes across all these documents?” that standard RAG cannot answer well.

A Real Use Case

Say you’re building an agent that needs to answer questions about a large legal case with hundreds of documents — witness statements, correspondence, expert reports, court filings. Standard RAG can answer “what did Witness A say on 14th March?” by finding the relevant chunk. It struggles with “what are the main areas of disagreement between the expert witnesses?” because that question requires understanding relationships across multiple documents, not finding a single passage.

GraphRAG handles the second question by having built a graph that connects the expert witnesses, their stated opinions, the topics they addressed, and their conflicts — and having community summaries that capture patterns across the whole collection.

Getting Started

Microsoft’s graphrag Python library is the reference implementation:

pip install graphrag

The basic workflow involves pointing GraphRAG at a directory of text files and running the indexing pipeline:

# Initialise project
python -m graphrag init --root ./my-project

# Configure settings.yml with your LLM endpoint (Azure OpenAI or OpenAI)
# Then run indexing
python -m graphrag index --root ./my-project

The indexing step uses a lot of LLM calls — it’s doing extraction and summarisation on every document and cluster. For a large document collection, the cost of the indexing step can be meaningful. This is one of the practical tradeoffs: GraphRAG’s global query capabilities come with higher indexing cost than standard vector embedding.

Once indexed, querying is straightforward:

import asyncio
from graphrag.query.cli import run_global_search, run_local_search

# Global search — questions about themes, patterns, aggregate insights
result = asyncio.run(run_global_search(
    config_dir="./my-project",
    data_dir="./my-project/output",
    query="What are the main themes across the expert witness reports?"
))

# Local search — specific factual questions
result = asyncio.run(run_local_search(
    config_dir="./my-project",
    data_dir="./my-project/output",
    query="What did Dr Smith say about the timeline?"
))

Where It Fits in Agent Pipelines

The natural place for GraphRAG in an agent pipeline is as a retrieval tool that the agent can call when the query requires cross-document synthesis. You might configure your agent with two tools: a standard vector search tool for point-in-time factual queries, and a GraphRAG tool for structural and thematic queries.

tools = [
    {
        "name": "vector_search",
        "description": "Find specific facts, passages, or answers in documents",
        "use_when": "specific factual query"
    },
    {
        "name": "graph_search", 
        "description": "Answer questions about themes, patterns, entities, relationships across multiple documents",
        "use_when": "synthesis, thematic, comparative, or structural query"
    }
]

Let the agent choose based on query type. For most queries, standard RAG is faster and cheaper. For queries that need the global view, GraphRAG earns its higher processing cost.

The Honest Limitations

GraphRAG isn’t the answer to everything, and it’s worth being clear about the tradeoffs.

The indexing cost is real. A large document collection — thousands of documents, millions of tokens — will run up a notable LLM API bill during indexing, and re-indexing when documents change adds ongoing cost. For frequently updated document sets, this can become expensive.

The pipeline also has latency. Global search queries aggregate across potentially hundreds of community summaries, which takes longer than a simple vector search. For real-time applications where response speed matters, this affects the user experience.

And the quality of the knowledge graph depends heavily on the quality of the entity and relationship extraction, which is itself an LLM call that can make mistakes. If the extraction step misses important entities or gets relationships wrong, the graph — and the queries built on it — will be correspondingly unreliable.

That said, for the specific problem GraphRAG is designed to solve — making sense of large, complex document collections where questions require understanding structure rather than finding passages — it’s genuinely effective. If you’ve hit the ceiling of what standard RAG can do for your use case, it’s worth evaluating.