Every provider in our catalog carries a quality score from 0 to 100. Exa is a 90, Serper is a 72, Reducto is a 92, Jina Reader is a 70. Those numbers drive real routing decisions: when you ask the router to sort by quality, the score is the sort key. So I owe you a straight answer about where they come from, and the straight answer is less scientific than you might hope and more useful than you might expect.
This post mirrors what lives at /docs/quality, with more room for the reasoning.
What the scores are
The scores are hand-curated editorial judgments. I read every provider’s docs, run their APIs, watch how they behave under our routing traffic, and assign a number. The inputs are qualitative: output fidelity for the category’s core job, feature depth, format cleanliness, how the API behaves at the edges, and how honest the documentation is about limitations. The scores are relative within a category, so a 90 in search and a 90 in parsing say nothing about each other.
Crucially, the scores are published in our open-source catalog, next to the prices. If you think a number is wrong, the file is right there, and the disagreement can happen in the open. I have adjusted scores based on exactly that kind of pushback, and I expect to keep doing so.
What the scores are not
They are not benchmark results. Nobody ran a thousand-document evaluation set through every parser and computed accuracy deltas to derive these numbers. When our catalog says Deepgram Nova-3 is an 88 and Groq’s Whisper is an 80, that is a considered judgment about product depth and output quality, not a measured word-error-rate gap. I refuse to dress editorial judgment up in benchmark clothing, because fake precision is worse than honest vagueness.
They are also not SLAs, uptime grades, or latency ratings. Reliability shows up in routing through a different mechanism entirely: the circuit breaker skips any provider whose error rate exceeds 30% over a five-minute window, based on live traffic rather than a static score. A provider having a bad day gets routed around automatically without me touching its quality number.
And they are not universal. Serper scores 72 not because it is mediocre but because it is exactly one thing: Google’s results via API. For a navigational query, Serper at 72 will serve you better than a neural engine at 90, and at a seventh of the price. The score compresses a multi-dimensional product into one integer, and compression loses information. That is the cost of having a sortable catalog at all.
Why hand-curated, and why I admit it
The tempting alternative is automated benchmarking, and it is genuinely on the roadmap: category-specific evaluation harnesses with published raw results, so the numbers can graduate from “Brian’s judgment” to “reproducible measurement”. I want that. But I have watched enough benchmark-driven leaderboards get gamed, go stale, or measure the wrong thing to know that a bad benchmark is more misleading than a labeled opinion.
The failure mode I most want to avoid is false authority. A score presented as objective gets trusted objectively. A score presented as curated invites exactly the right amount of skepticism: use it as a strong prior, override it where your workload disagrees. In my experience running the router, the ordering within a category is right far more often than not, and the ordering, not the absolute values, is what routing consumes.
There is one more honest reason: some categories resist benchmarking. Voice quality in text-to-speech is famously subjective; ElevenLabs scores 93 in our catalog largely because, qualitatively, that is where the market and my own ears agree the ceiling is. Pretending a benchmark produced that number would be theater.
How the scores interact with routing
The router supports sorting by price, speed, or quality. Cheapest-first is the default, and quality scores act as a filter and tiebreaker: you can set a quality floor so cheapest-first never routes below a threshold you trust, which turns the score from a ranking into a guardrail. Quality-first sorting inverts it, useful for user-facing output where the marginal dollars are cheaper than the marginal embarrassment. Both are one-line changes in your saved routing preferences rather than code edits.
What the scores never do is hide anything. Every response includes the full routing.attempted chain and the exact price charged, so when the router picks a provider on your behalf, you can see the pick, the fallbacks, and the cost. A curated score influencing a transparent decision is a very different thing from a black box.
How to use them (and when to ignore them)
Treat the scores as a prior, not a verdict. If you are entering a category cold, sorting by quality tells you where the ceiling is and price tells you where the floor is, and the gap between them tells you whether the category is worth tiering. Once you have real traffic, your own success rates outrank my numbers, always. The scores exist so your first routing decision is informed; your hundredth should be informed by your own data.
Every score, every price, and the reasoning behind the methodology are public: start with the search API comparison to see the scores in context.