Skip to content
Journal / scrape

Firecrawl for agent builders: scraping, search, and failover fit

Firecrawl does JS-rendered scraping at $1.92 per 1k pages routed and search with page content at $2.40. Where it fits in an agent's failover chain.

Firecrawl occupies an unusual spot in our catalog: it is the only provider that shows up in two categories. Its scraping endpoint is the quality leader of our scrape category, and its search endpoint is the budget option among content-included search APIs. Since I route traffic to both and pay for both, this is my working guide to what Firecrawl is, what it costs, and where it belongs in an agent’s toolchain.

The scrape endpoint: the quality tier

Firecrawl’s core product takes a URL and returns the page as clean, LLM-ready markdown. Two things distinguish it within our scraping category.

First, JS rendering. A large share of the modern web does not exist until JavaScript runs: SPAs, dashboards, infinite-scroll pages, content behind client-side hydration. A plain HTTP fetch of those pages returns a shell. Firecrawl renders in a real browser environment, so the markdown you get back reflects what a human would see.

Second, output quality. The point of scraping for agents is stuffing pages into context windows, and the cleaner the markdown, the fewer tokens you burn on nav bars and cookie banners. In my experience this is where Firecrawl earns its catalog score of 85, the highest in our scrape category.

The price for all this: $1.92 per 1k pages routed (routed prices are provider list plus 20%, per the pricing docs). That is roughly 5x the budget tier, where Jina Reader and Spider both sit at $0.36 per 1k routed, scored 70 and 72. Spider also does JS rendering at that price, which makes it the interesting middle case: rendering without the premium. The full three-way breakdown is in my scraping API comparison, and live prices are on the scraping comparison page.

The search endpoint: search plus content in one call

Firecrawl Search is the newer surface, and its pitch is structural: results can come back with the page content included, not just links and snippets. For agents this collapses the classic two-step loop (search, then scrape the top results) into a single call.

At $2.40 per 1k queries routed it is the cheapest content-included search in our catalog; Exa, the other provider that returns text with results, costs $8.40. Our score for Firecrawl Search is 74, which is honest about what it is: utilitarian results, not the neural-retrieval quality of the premium tier. But price the workflow rather than the call. If a Serper query at $1.20 per 1k triggers three scrape calls afterwards, the “cheap” path costs more than one Firecrawl Search call that arrives with content attached, and it is slower, since the scrapes happen sequentially after the search returns. For read-heavy research agents, that arithmetic is worth actually doing with your own traffic shapes. The category-wide numbers are on the search comparison page.

Where Firecrawl sits in a failover chain

This is the router’s-eye view, and I think it is the most useful lens.

Scraping, cheapest-first. The default chain I run: Jina or Spider takes the volume at $0.36, Firecrawl is the escalation. The budget tier handles well-behaved pages fine, and when it returns an error, the router fails over. One hard-won detail: an empty or near-empty body from a scraper should be treated as a failure, not a success, because a blocked or unrendered page frequently comes back as a 200 with nothing in it. A chain that only fails over on HTTP errors will happily deliver you empty markdown. When the escalation does fire, Firecrawl’s rendering and block-resistance make it exactly what you want as the last resort: the provider most likely to succeed on the pages that broke the cheap tier.

Scraping, quality-first. For agents where a bad extraction poisons downstream reasoning (RAG ingestion, anything a model will treat as ground truth), invert the chain: Firecrawl first, budget tier as failover. You pay 5x on the happy path and buy the highest-probability-of-clean-content first attempt.

Search. Firecrawl Search slots in as the mid-chain option when content-included results matter but Exa’s price does not fit, or as failover behind Exa for research agents, since both return text and the normalized schema means the swap is invisible to your code.

On our router, all of this is configuration rather than architecture: failover tries up to 3 providers, a circuit breaker skips any provider above a 30% error rate over 5 minutes, and saved routing preferences flip between cheapest-first and quality-first without touching agent code.

Honest caveats

Firecrawl is the premium scrape pick, not the universal one. At $1.92 per 1k, high-volume scraping of simple, static pages is paying for a browser render you do not need; that traffic belongs on the budget tier. And the search endpoint’s 74 score is real: for semantic research queries where result quality is the whole game, Exa and Perplexity are scored meaningfully higher in our catalog. Firecrawl’s strength is being very good at the intersection (render-heavy pages, content-included search, one vendor covering two categories), and that intersection happens to sit dead center of what agent builders need.

Current prices and scores for every provider mentioned here are on the scraping API comparison page.