TL;DR:

  • Haystack 2.x uses a component-based Pipeline architecture — wire together Embedders, Retrievers, Generators, and DocumentStores to build RAG and search applications
  • Native integrations with OpenAI, Anthropic, Cohere, HuggingFace, Qdrant, Weaviate, Elasticsearch, Pinecone, and more — swap components without rewriting your pipeline
  • The declarative YAML pipeline format makes deployments reproducible and lets you version your pipeline configuration alongside your code

If you’ve spent any time building RAG applications in Python, you’ve probably encountered the same friction: stitching together an embedding model, a vector store, a retrieval step, and an LLM into something that reliably works in production. Haystack, built by deepset, is a framework specifically designed to handle that complexity. The 2.x rewrite cleaned up a lot of rough edges from the earlier version and introduced a pipeline model that’s genuinely pleasant to work with.

How Haystack Pipelines Work

The core concept is straightforward. A Pipeline is a directed graph of Components. Each Component has defined inputs and outputs, and you connect them by specifying which output of one component feeds into which input of the next.

A minimal RAG pipeline might connect:

  1. A TextEmbedder — takes a query string and returns an embedding vector
  2. An InMemoryEmbeddingRetriever — searches a document store by vector similarity and returns relevant documents
  3. A PromptBuilder — formats the retrieved documents and the original query into a prompt
  4. An OpenAIGenerator — sends the prompt to an LLM and returns the response
from haystack import Pipeline
from haystack.components.embedders import SentenceTransformersTextEmbedder
from haystack.components.retrievers import InMemoryEmbeddingRetriever
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator
from haystack.document_stores.in_memory import InMemoryDocumentStore

document_store = InMemoryDocumentStore()

rag_pipeline = Pipeline()
rag_pipeline.add_component("embedder", SentenceTransformersTextEmbedder())
rag_pipeline.add_component("retriever", InMemoryEmbeddingRetriever(document_store=document_store))
rag_pipeline.add_component("prompt_builder", PromptBuilder(template=template))
rag_pipeline.add_component("llm", OpenAIGenerator())

rag_pipeline.connect("embedder.embedding", "retriever.query_embedding")
rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder.prompt", "llm.prompt")

You run the pipeline by calling rag_pipeline.run({"embedder": {"text": query}}). Haystack handles routing the outputs through the graph.

Document Stores

The DocumentStore is where your indexed content lives. Haystack has native integrations for:

  • InMemoryDocumentStore — great for development and small datasets; no infrastructure needed
  • ElasticsearchDocumentStore — full-text and vector search in Elasticsearch 8.x
  • QdrantDocumentStore — Qdrant vector database with efficient approximate nearest neighbour search
  • WeaviateDocumentStore — Weaviate’s hybrid BM25 + vector search
  • PineconeDocumentStore — managed serverless vector search

Swapping the DocumentStore doesn’t change your pipeline code. You configure the store, pass it to the Retriever component, and the pipeline is otherwise identical. This is useful if you start with InMemory for development and need to move to Qdrant for production without rewriting anything.

Indexing Pipelines

You need two pipelines: one to index documents, one to query. The indexing pipeline converts raw content into embedded document chunks and stores them.

A typical indexing pipeline:

from haystack.components.converters import TextFileToDocument
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.embedders import SentenceTransformersDocumentEmbedder
from haystack.components.writers import DocumentWriter

indexing = Pipeline()
indexing.add_component("converter", TextFileToDocument())
indexing.add_component("splitter", DocumentSplitter(split_by="sentence", split_length=5))
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
indexing.add_component("writer", DocumentWriter(document_store=document_store))

indexing.connect("converter", "splitter")
indexing.connect("splitter", "embedder")
indexing.connect("embedder", "writer")

DocumentSplitter handles chunking — you can split by word, sentence, or passage, and configure overlap. Getting chunk size right matters: too small and you lose context, too large and retrieval becomes imprecise. Sentence-level splitting with overlap of a few sentences is a solid starting point.

LLM Integrations

Haystack has first-class integrations for the main LLM providers. The Generator component handles the LLM call:

  • OpenAIGenerator — GPT-4o, GPT-4.1, o3, any OpenAI-compatible endpoint
  • AnthropicGenerator — Claude 3.5/4.x models
  • CohereGenerator — Cohere Command R/R+
  • HuggingFaceLocalGenerator — local model inference via transformers

If you need streaming responses, there are async-capable variants (OpenAIChatGenerator with streaming_callback). For conversational RAG where you need to maintain chat history, the Chat variants manage the message list for you.

Hybrid Retrieval

Pure vector similarity search has a known limitation: it can miss exact keyword matches that a traditional BM25 index would catch. Haystack supports hybrid retrieval that combines dense vector search with sparse keyword search and merges the results.

The InMemoryBM25Retriever and InMemoryEmbeddingRetriever can run in parallel within a pipeline branch, and a DocumentJoiner component merges and re-ranks their outputs before the prompt step. For production workloads where precision matters, hybrid retrieval reliably outperforms either approach alone.

YAML Pipeline Serialisation

Haystack pipelines can be serialised to YAML, which makes them portable and versionable:

with open("rag_pipeline.yaml", "w") as f:
    rag_pipeline.dump(f)

loaded = Pipeline.load("rag_pipeline.yaml")

The YAML format captures the full pipeline graph including component configuration. Teams can version their pipeline definitions in git, run the same pipeline in development and production from the same config file, and swap components by editing YAML rather than code.

When Haystack Makes Sense

Haystack is strongest in a few specific scenarios. If you’re building document Q&A, enterprise search, or knowledge base chatbots — anything where the core workflow is “find relevant content, generate a response” — it gives you a solid production foundation without building retrieval infrastructure yourself.

It’s also a good fit if you want to stay provider-agnostic. Building directly on the OpenAI SDK makes it harder to evaluate alternatives later. Haystack’s component abstraction makes swapping embedding models or LLM providers a config change rather than a code rewrite.

Where it’s less obviously a fit: if your agent workflow is primarily about tool use and multi-step reasoning rather than document retrieval, something like LangGraph or the Anthropic Agents SDK might be a better match. Haystack is retrieval-first; the agent orchestration support exists but isn’t the framework’s primary strength.

The documentation at docs.haystack.deepset.ai is thorough and includes working cookbook examples for the common patterns. It’s worth starting there before writing any pipeline code.