What Moves a RAG System's Score, and What Doesn't

On 20 October we spoke at Cloudflare Connect at Moscone West in San Francisco. The talk covered two months of retrieval experiments: public benchmarks, a synthetic legal corpus, more than a hundred websites and a live customer deployment. We went in believing that routing each question to the right retrieval method would beat any single method. It did not. Most of what we expected to matter did not, and two things we had not looked at closely mattered a great deal.

The short version

1. Whether the answer is in the index matters more than how you search it.
2. Routing each question to the “right” retrieval method looks powerful in hindsight, but we could not capture that gain in practice.
3. The best setting depends on the documents. Our own default lost on public benchmarks, and the winner changed again on a legal corpus.
4. A higher score is not always a better answer. One model swap raised our suite score ten points and changed nothing a visitor would notice.
5. So measurement has to come first: test on the customer’s own content, against a pass bar set before the run.

Every figure below comes from our research notebooks. Where we have a 95% confidence interval, it sits next to the number in square brackets.

One setup, very different scores

We build a demo assistant for each prospect’s website and score it on that site’s own test questions. Almost every one of those demos runs the same default retrieval setup. The scores are nowhere near the same.

That spread is the starting point for everything else. If the same system scores 0.59 on one site and 0.99 on another, the first question is what differs about the site.

Fixing coverage beat changing the stack

The largest single gain we have measured came from making sure the needed pages were actually in the index. On sites where the crawl had missed pages, ingesting them raised scores by 31 points. Swapping the entire retrieval stack moved the mean from 84.3% to 84.0%.

A retrieval system cannot return a page it never ingested, and no amount of reranking recovers it. We now check coverage before anything else.

Within one corpus, the setting does matter

None of this means retrieval settings are irrelevant. On a fixed corpus with the answers present, the choice changes the score, and keyword search on its own came last on both public benchmarks we used.

Our own default weighting lost

That result was uncomfortable, because the hybrid search we shipped was weighted 70% toward keywords. On both benchmarks a weighting of about 5% keyword did better.

On a legal document set, the ranking flipped

Then we ran the same seven settings on a different kind of document: a synthetic law-firm corpus of 9,288 files with 242 retrieval tasks. The setting that won on science fell to sixth. An equal weighting of keyword and meaning came first.

So “which weighting is best” has no single answer. It depends on the documents, which is the argument for testing on each customer’s own corpus instead of shipping one global default.

Per-question routing: a large ceiling we could not reach

If the best method varies, the obvious next step is a router that picks the method per question. On 150 held-out questions from a live customer deployment, two retrieval backends disagreed on 35 of them.

We did not stop at one attempt. Across eight routing experiments and more than 23 router configurations, on the customer set, public benchmarks, the legal corpus and one website, none beat the best fixed pipeline on held-out data. The best-looking gain, +0.051, shrank to +0.030 [−0.023, +0.074] against a plain baseline, an interval that includes zero.

The practical reading: pick a good fixed setting for the corpus and spend the effort on coverage.

A score is not an answer

One more result changed how we read our own numbers. We swapped the answering model on 28 sites, keeping the index and the judge fixed. The suite score rose about ten points. Then we asked ordinary visitor questions on two of those sites, and the two models did the same.

Before trusting a score, check what the test suite is measuring.

Seeing a corpus

Part of why the corpus dominates is that corpora have very different shapes. Plotting one makes that visible. This is a sample of the SciFact collection, one dot per abstract.

Five checks before trusting a RAG architecture claim

Each check comes from one of the findings above. They are also how we now run QA for customers.

1. Check coverage first: is the answer in the index at all?
2. Measure on your own content, comparing setups on the same questions.
3. Write down the pass bar before you look at the scores.
4. Report by site and by question type, with the noise next to every number.
5. Treat a hindsight “oracle” as a ceiling, never as a result.

Short videos

Each finding has a two-minute video:

• We tried routing eight ways. None beat one fixed pipeline.
• Fixing coverage was worth 31 points
• Our own default fusion weight lost

Three more, on the website spread, the legal flip and the five checks, come out on our YouTube channel between 12 and 14 November.

What these results are not

The answer scores come from an AI judge we have not yet calibrated against human raters, and the results are exploratory. None of them is a signed TrustBench result. When a finding here becomes one, it will be on the public leaderboards with a manifest anyone can verify.