One of the most underestimated problems in building AI agents is the gap between “information is on the web” and “information is in a format my LLM can use.” Web pages are built for browsers — JavaScript-rendered content, navigation menus, cookie banners, footers, ads, tracking scripts. Feed a raw HTML page to an LLM and you’re wasting context on noise.

Firecrawl solves this. It’s a web scraping and crawling API that converts web pages into clean markdown (or structured JSON) that’s actually usable in LLM contexts. It handles JavaScript rendering, extraction of the main content, and provides both single-page and multi-page crawling. For AI agent developers who need reliable web access as part of their tool stack, it’s become something of a default choice.

What Firecrawl Does

At its core, Firecrawl takes a URL and returns clean, LLM-ready content. The default output is markdown — headings, paragraphs, tables, and code blocks preserved, navigation and chrome stripped. You can also get structured JSON if you provide a schema for what you want to extract.

The API has three main modes:

Scrape (/scrape): Fetch and convert a single page. The most common use case for agent tools. Give it a URL, get back markdown.

Crawl (/crawl): Traverse an entire site from a starting URL, respecting depth limits and URL patterns you specify. Useful for building RAG knowledge bases from documentation sites.

Map (/map): Return all URLs discoverable from a starting point without fetching the content. Useful for planning what to crawl before you crawl it.

The JavaScript rendering is worth calling out specifically. Most web scraping libraries operate on raw HTML and fail on content that’s loaded dynamically. Firecrawl runs a headless browser, so you get what a user would actually see, not the raw document skeleton.

Getting Started

Installation is a single package:

pip install firecrawl-py

A basic scrape looks like this:

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="your-key")

result = app.scrape_url(
    "https://docs.anthropic.com/en/docs/about-claude/models/overview",
    formats=["markdown"]
)

print(result.markdown)

That returns the full page content as clean markdown, ready to pass directly into an LLM context or store in a vector database.

Using Firecrawl as an Agent Tool

The most common pattern is wrapping Firecrawl as a tool that your agent can call when it needs to retrieve web content. In practice, this looks like:

from firecrawl import FirecrawlApp
from your_agent_framework import tool

app = FirecrawlApp(api_key="your-key")

@tool
def fetch_webpage(url: str) -> str:
    """Fetch and return the content of a web page as clean text."""
    result = app.scrape_url(url, formats=["markdown"])
    return result.markdown[:8000]  # trim to context budget

The context budget trim is important. Firecrawl returns good content, but long pages can still blow your token budget. A practical approach is to either truncate at a sensible limit, chunk the content and pass relevant sections, or use Firecrawl’s extract mode with a schema to pull only what you need.

Structured Extraction

The extract mode is genuinely useful for agentic use cases where you want structured data rather than prose. You define a Pydantic schema and Firecrawl uses an LLM extraction step to populate it from the page content:

from pydantic import BaseModel

class ProductInfo(BaseModel):
    name: str
    price: float
    rating: float | None
    availability: str

result = app.scrape_url(
    "https://example.com/product/123",
    formats=["extract"],
    extract={"schema": ProductInfo.model_json_schema()}
)

product = result.extract  # typed ProductInfo object

This is a good pattern for agents doing research across multiple pages where you want to accumulate structured information rather than raw text.

Crawling for RAG Knowledge Bases

If you’re building a RAG pipeline over an entire documentation site or knowledge base, the crawl endpoint is the right tool:

crawl_result = app.crawl_url(
    "https://docs.yourproduct.com",
    limit=100,
    scrape_options={"formats": ["markdown"]}
)

# crawl_result.data is a list of page results
for page in crawl_result.data:
    # chunk and embed each page
    store_in_vector_db(page.markdown, metadata={"url": page.metadata["url"]})

The limit parameter controls how many pages to crawl. You can also filter by URL patterns to stay within relevant sections of a site.

Firecrawl vs. the Alternatives

The main alternatives are BeautifulSoup (raw HTML parsing), Playwright/Selenium (headless browser control), and Apify (managed scraping platform).

BeautifulSoup is fine when the content is in the raw HTML and you know exactly what selectors to target. It’s completely wrong for agent use cases where you’re fetching arbitrary URLs — you can’t write a selector in advance for a page you haven’t seen.

Playwright gives you full browser control but you’re writing the extraction logic yourself. For agentic use cases where the agent decides what to fetch, you’d need to wrap Playwright with your own content extraction, which is what Firecrawl already does.

Apify is a full enterprise scraping platform with its own marketplace of scrapers. If you need to scrape specific sites at scale (Twitter, LinkedIn, e-commerce platforms), Apify’s pre-built scrapers are excellent. For general-purpose web access in an LLM agent, Firecrawl is simpler.

Exa is the other tool worth mentioning. Where Firecrawl fetches content from URLs you provide, Exa is a search API that returns semantically relevant results. The two complement each other: Exa to find what to fetch, Firecrawl to fetch it cleanly.

Practical Limits to Know

Rate limits depend on your plan tier. The free tier is useful for development but hits limits quickly in production. Budget for API costs if your agents are fetching pages frequently — web scraping at scale adds up.

Some sites block headless browsers via Cloudflare or similar anti-bot systems. Firecrawl handles many of these cases, but it’s not magic — heavily protected sites may still return errors. For sites that actively resist scraping, Firecrawl has an actions parameter that lets you specify browser interactions (clicks, waits, scrolls) before capture.

Content that requires authentication also requires you to handle session management separately. Firecrawl supports passing cookies, which is usually sufficient for logged-in scraping of sites where you have legitimate access.

Self-Hosted Option

Firecrawl is open source (MIT) and can be self-hosted if you’d rather not send content through the API. The self-hosted version requires a bit more setup (Redis, Playwright, the main service) but is well-documented and gives you full control over data handling. For agents processing sensitive internal content, the self-hosted option is worth the additional complexity.

If your agents need to read the web, Firecrawl is the shortest path from a URL to useful LLM context. The API is clean, the output quality is high, and the integrations with most agent frameworks are straightforward.