Scraping is the least reliable category I route, and I mean that as an observation, not an insult. Search APIs answer from their own index. Scraping APIs have to go fetch a page from someone else’s server, and that someone is often actively trying to stop them. The result is a category where failure is a normal operating condition, and where the worst failures do not look like failures at all.
This post is about the failure taxonomy I have come to rely on, and the failover patterns that follow from it.
The failure modes, from loud to silent
The loud ones are easy. A 403 means you were blocked outright. A 429 means you were rate limited. A timeout means the target was slow or the scraper gave up. All of these arrive as errors, your code sees them, and any retry logic you have gets a chance to run.
The dangerous failures return a 200. The most common is the empty or near-empty body: the scraper connected, got served a shell page, and dutifully converted nothing into markdown. Close behind is the JS wall, where the real content only exists after client-side rendering, so a non-rendering scraper returns a page skeleton and a “please enable JavaScript” notice. Then there are consent walls, cookie banners, bot-check interstitials, and login gates, all of which produce plausible-looking HTML that contains none of the content you asked for.
For an agent, silent failures are worse than errors. An agent that receives an error can retry or reroute. An agent that receives an empty string treats it as a valid observation, reasons about it, and confidently reports that the page had no relevant information. The garbage propagates.
Rule one: empty content is a failure
The single highest-leverage change you can make to a scraping pipeline is to stop trusting HTTP status codes. Define success by the content itself: a minimum character count is a crude but effective floor, and checking for actual markdown structure (headings, paragraphs, links) catches shell pages that pad their length with boilerplate. If a page fails those checks, treat it exactly like a 403 and move on to the next provider.
This is how our router behaves internally, and it is why I keep insisting on the framing: a 200 with no content is a failed scrape. Bill it, log it, and retry it accordingly (on our side, failed calls are not billed, which keeps the incentives honest).
Why failover works better here than retries
Retrying the same scraping provider against the same page mostly reproduces the same result. The block that stopped the first attempt is still there. What actually changes the outcome is trying a provider with different infrastructure: different proxy pools, different rendering behavior, different fetch strategies. This is the same argument I made in why AI agents need failover, but scraping is the category where it applies most literally, because the failures are caused by the target site’s countermeasures, and different scrapers trip different countermeasures.
Our scrape catalog has a convenient shape for this. Jina Reader costs $0.36 per thousand pages routed with a quality score of 70. Spider is also $0.36 routed, scores 72, and does JS rendering. Firecrawl costs $1.92 routed, scores 85, and also renders JS. (Routed prices are list plus 20%, per the pricing docs.)
That gives you a natural escalation ladder: start cheap with Jina, fall through to Spider when the page needs rendering or Jina gets blocked, and reserve Firecrawl as the closer, the provider most likely to succeed when the first two failed, at about five times the price. Since the expensive provider only sees the hard residue of your traffic, your blended cost stays close to the cheap tier while your success rate approaches the expensive one.
The mechanics worth stealing
If you build this yourself, three details matter beyond the provider order.
First, cap total attempts. Our router tries at most three providers per request, because a page that defeated three different scraping stacks is overwhelmingly likely to defeat a fourth, and agents should get a clean failure rather than an endless spinner.
Second, track provider health across requests, not just within them. A circuit breaker that skips any provider running above a 30% error rate over the last five minutes stops you from feeding traffic into a provider mid-outage, and lets it back in automatically when it recovers.
Third, keep the attempt history. Every response from our router includes the full routing.attempted chain, which sounds like a debugging nicety until the first time you need to answer “why did this scrape cost 5x more than usual” and the answer is visible in one field: two cheap providers failed on that domain, the closer succeeded. If a specific site consistently ends up at the expensive provider, a saved routing preference can send it straight there and skip the doomed attempts.
None of this makes scraping reliable in the absolute sense. It makes scraping fail cleanly, cheaply, and observably, which is the realistic goal.
Current prices and quality scores for every scraping provider we route are side by side on the scraping API comparison page.