What a Benchmark Has to Prove About Itself

The most important number on either of our public leaderboards is 0.078.

It is not the best score. It is the worst one, and it is the only row that makes the others mean anything. Same model, same sixty questions, same judge — retrieval switched off. Without the corpus the model scores 0.078, which is to say it cannot answer. Every other row on that board is therefore measuring retrieval rather than what the model already happened to know.

Almost no published RAG benchmark carries that row. It costs one extra run and it is the difference between a number and a measurement.

Both boards are live on the public TrustBench leaderboards, alongside four others. Every row there carries a link to the signed manifest that contains it, and the id of the key that signed it.

Update, 26 September: the nutrition board now has rows built on Jev, a ranking model from TypeSafe. They include the highest score the board has recorded (0.947), a controlled ablation showing that the lead comes from Jev (+0.130 on paired questions), and a test of Jev as a judge (94.7% agreement with Gemini, and no self-preference). Read the results.

The thing a signature does not prove

In April 2026, a team at UC Berkeley’s RDI — Hao Wang, Qiuyang Mang, Alvin Cheung, Koushik Sen and Dawn Song — adversarially stress-tested eight of the most cited AI agent benchmarks, across more than 3,556 tasks. Seven of the eight were driven to roughly 100%. OSWorld held out at 73%.

The exploits are worth reading in full, because they are not clever:

• SWE-bench Verified — a conftest.py of ten lines of Python “resolves” every instance.
• WebArena — point Chromium at a file:// URL and read the gold answer out of the task config. Roughly 100% across all 812 tasks.
• Terminal-Bench — a fake curl wrapper scores all 89 tasks without a line of solution code.
• FieldWorkArena — the exploit is one message: {}. Validation checked only that the last message came from the assistant. It never compared the answer to ground truth.

RDI’s framing is the sentence to keep: the evaluation was not designed to resist a system that optimizes for the score rather than the task.

Now the part that matters for anyone building attestation. Cryptographically signing those results would have caught none of them. Every one of those runs was honest. The harness did exactly what it was told. The outputs hash matches. The signature verifies. A signed manifest over a gamed benchmark is a perfectly valid signature over a meaningless number — and it is worse than an unsigned one, because it carries an air of proof it has not earned.

This is the trap in front of everyone selling “verifiable evals” right now, us included. Signing the result is the easy link. It is also the one that proves the least.

Four signatures, not one

An evaluation is a claim about a tuple: this model, on this dataset, through this harness, produced this result. Four things, four places to lie or err, four manifests.

The honest position on that diagram is that we ship the rightmost box and part of the one beside it. The two on the left are roadmap. We would rather publish the diagram with the gaps drawn in than let the word “attested” do work it cannot do.

An attestation that grades its own strength

Here is the design decision this whole post is really about.

A TrustRun manifest carries a field, results.provenance.sourceKind, with three values. The open-source verifier — MIT, no dependency on anything we control, runnable against a manifest and its outputs with no account — reports on it.

Our two retrieval boards sit on the middle rung. The scores were computed by Divinci’s scored-QA pipeline and republished into a signed TrustRun; the TrustBench harness did not invoke the model itself. Anyone who runs the verifier against those manifests gets told so, in these words:

results.provenance.sourceKind is 'republished': the score was computed by an earlier run, not by the evaluator execution this manifest attests

We could have left the field out. The signature would still be valid, the boards would look identical, and approximately nobody would have asked. Instead the manifest volunteers the weakness, the verifier surfaces it, and strict: true turns it into a hard failure for any consumer who wants the stronger bar.

The stub rung exists for a reason that still bothers me. A transport fixture — a manifest whose score is a hash of two id strings, whose model was never called — verified identically to a real evaluation, on two live public URLs. The signature was genuinely valid, because it was genuinely our signature over a genuinely unmodified document. Nothing was broken. The format simply had no way to say “this is not a measurement,” so it said nothing, and silence read as assent.

That is the general lesson, and it is not about cryptography at all: a verification format that can only say “valid” will be used to imply things it never checked. The useful ones grade their own strength and let the reader set the bar.

Footnote: we checked whether the judge mattered

A fair objection to everything above: both boards were scored by gemini-2.5-flash, which is a Google model ranking a board where Google’s Vertex retrieval comes first. And we could not point to evidence that judge is any good — in our own calibration table it is the anchor, correlation 1.0 with itself and measured against nothing. Being the ruler is not the same as being accurate.

So we re-scored every published row with a second judge from a different family, @cf/deepseek-ai/deepseek-v4-flash-0731, against the same stored answers — generation held fixed, the judge the only thing that moved. 420 judge calls, no failures.

Both boards rank identically, and no retrieval row moves by more than 0.016:

The ordering is not an artifact of who judged it. But look at the last row, because it is the more interesting result: agreement is not uniform across the score range. The two lowest-scoring rows are the two where the judges agree least — the no-retrieval control disagrees on 17 of 60 items, against 2 of 60 for the top row. Judges concur on answers that are clearly grounded and diverge on bad ones.

The practical consequence is a correction to how the headline should be read. The gap between retrieval and no retrieval is robust. The baseline’s exact value is not — deepseek scores ungrounded answers about four points more generously. Cite the gap, not the floor’s third decimal.

And none of this shows either judge is right. Two judges agreeing is agreement, not correctness; both can be wrong in the same direction. Only calibration against a human anchor settles accuracy, and that is still open — which is itself the honest version of a claim most leaderboards never make at all.

Three ways a score quietly stops meaning anything

Building these boards turned up three failures that share a shape. None of them produced a wrong-looking number.

A measurement loses its identity. A row on a retrieval board claims: this model, retrieving through this stack, scored X. We had a publication path that averaged every result the suite had ever produced for that release — every replicate, every earlier sweep, every other retrieval backend the release had been tested against — and labelled the mean with whichever stack the release pointed at that day. It renders like any other row. Nothing about it looks wrong. The identity of a measurement lives in the document or it does not live anywhere, and a label is not a document.

Errors with a direction survive review. Our grader had a rule that failed any answer mentioning a passage id other than the expected one. Correct about hedging: naming every passage is not attribution. Wrong about the opposite case — a model that cites the right passage and explains why another does not apply was marked wrong. That is better attribution than a bare citation.

What makes this class dangerous is the direction. The error only ever suppressed passes, so scores moved the way a fix intends and the board looked stricter rather than broken. It was the third instance of that exact shape in one scorer. And every time, it marked down the models that explain their reasoning — which is the behaviour a buyer most wants. An eval whose errors are all conservative is not “safe”; it is systematically biased against legibility, and you cannot see it in the aggregate because the aggregate moves correctly.

Reproducibility is a property of the spec, not the code. The grading rules that matter — what counts as declining, as attribution, as ruling a passage out — travel inside the signed benchmark spec, so a third party reproduces our scores without running our software. We once added such a rule to the corpus, to both graders, and to the type, with every test passing, and it reached production in none of those forms: the serializer that builds the signed spec enumerated the fields it copied, and the new one was not on the list. Correct code, correct tests, a rule that could not fire on a single run.

Reading the code proved nothing. Asking the API what it actually serves settled it in one request. That habit — ask what the system serves, not what the repository contains — has caught more of these than any test we have written.

The next link is the dataset, and we have our own mistake as the argument

Our first attempt at the nutrition board used an auto-generated question set. Thirty-five of its fifty questions referred to an unnamed speaker or scene: “What event were the speakers attending last night?” Those do not measure retrieval quality. They measure which transcript chunk a backend happened to surface.

Worse, and more subtly: the generator draws each expected answer from one release’s own RAG pipeline. The benchmark was quietly a measurement of the system that produced it.

No signature over the result would ever have caught that. The runs were honest, the harness correct, the outputs hashed, the signature valid. The benchmark was simply born biased. The only thing that catches it is provenance over the dataset — which is why item-level content credentials are the next link and not a nice-to-have.

We replaced the set with hand-authored questions where every reference answer is pinned to a verbatim evidence substring of a real chunk, machine-checked. For the nutrition board the evidence had to be present in all four backends’ copies of the corpus, or the comparison would be measuring ingestion differences and calling it retrieval. That constraint had a real cost, which the board discloses: eighteen of twenty-two book questions come from a single book, the only one whose body text survives in all four ingestions.

All of that was enforced by hand, by people, once. A dataset manifest is what turns it into something a stranger can check. Four questions it would make answerable, none of which a result signature can touch:

1. Who wrote this item, and were they qualified to?
2. Was the expected answer derived from a system under test?
3. Is the supporting evidence present in every corpus being compared?
4. Has this item leaked into training data?

The fourth is the one with the longest shadow. Contamination is currently argued about statistically, after the fact, with detectors. With per-item credentials it becomes a provenance query with an answer. C2PA’s 2.4 specification already ships AI and ML guidance, which means the dataset link has a live standard to conform to rather than a new one to invent — and we would rather adopt than invent. The same goes for W3C Verifiable Credentials for author identity and SLSA’s language for pipeline attestation.

Why this is not a 2030 argument

The usual objection to all of this is that it is infrastructure for a regulatory regime that has not arrived.

Partly, it already has. The EU AI Act has applied in general since 2 August 2026, and its general-purpose AI chapter has been in force since 2 August 2025 — which obliges providers of models with systemic risk to perform model evaluations, assess and mitigate risks, and keep technical documentation. That is an evaluation duty, live now.

The heavier documentation duty is dated rather than here. Annex IV — dataset documentation, validation reports, change logs, version control maintained across the lifecycle, which is near enough the manifest chain above written by lawyers instead of engineers — attaches to high-risk systems, and that schedule was amended: Article 6(2) and Annex III systems now fall due 2 December 2027, and Annex I product-embedded ones 2 August 2028. Anyone telling you Annex IV landed in August 2026 is reading the original timetable. We had it wrong here too until we read Article 113 instead of a summary of it.

Two and a bit years is not long to build an evidence chain that a regulator will accept, which is the actual argument for starting now.

Meanwhile the market it has to police runs largely on self-report. We went looking for a hard number — a widely repeated figure says only one of the hundred models on a major public leaderboard carries an independent verification badge — and could not stand it up: the site in question publishes no verification indicator at all today, so there is nothing to count. Quoting an unverifiable statistic in a post about verifiability would have been a nice way to refute ourselves.

The structural point survives without it, and matters more. A number from the company selling the model is a claim. The same number reproduced on a standardized harness is evidence. And because every vendor evaluates through its own scaffold — its own tool definitions, retry logic, context management, prompting — a self-reported score silently bakes that scaffold in.

Which leads somewhere we did not expect when we started, and is probably the most useful thing in this post.

Everyone benchmarks models. Nobody benchmarks the scaffold. Yet RDI’s result says the scaffold decides the score, and self-reporting says the scaffold is invisible in nearly every published number. The missing instrument is a board that holds the model fixed and varies one coordinate of the stack — retrieval backend, chunking strategy, embedding model, reranker, parser, judge, guardrail, agent harness.

That is what the two boards at the top of this page are. It is also why the 0.078 row matters so much: in a component benchmark, the negative control is not a courtesy. It is the axis.

Update, 26 September: PixelRAG + Jev, and what its lead is made of

Since this post first went up, the nutrition board has gained four rows. Two of them are built on Jev, a model from TypeSafe that does not generate text and returns a probability for each option it is asked about. One is the same retriever with Jev removed, run as a controlled ablation. The fourth belongs to another workspace and is not discussed here. Everything this post argues about the original four rows applies to these, and they also give it its clearest test.

The system. PixelRAG renders every page of a corpus to an image, cuts each page into overlapping tiles, and retrieves tiles with a vision embedding model (Qwen3-VL-Embedding-2B). We paired it with keyword search over the text printed on each tile and gave the union of both candidate lists, up to sixty tiles, to Jev. Jev is asked does this tile state the answer to the question?, and the tiles reach the answering model in order of its probability. The answering model, the sixty questions and the judge are the same as for every other row, and the corpus is the same five collections, with one difference in PixelRAG’s favour: for the eighteen PDFs it indexes every rendered page, where the other rows hold only the passages their parser kept.

Results

Each score is the median of three signed runs. The interval is a bootstrap over the sixty questions of the run shown, and it is what the public board now prints under every score. The last row was scored by a different judge, so the board ranks it separately: the board now gives each judge its own table, and does not compare scores across them.

PixelRAG + Jev is the highest score the board has recorded, and it holds that position under every judge we have applied. What the sixty questions can establish about it is narrower, and it is best seen question by question. Every row answers the same questions, so the right comparison is the paired difference between two rows on each question, not the overlap of two separate intervals:

Four results follow.

1. The top of the board is a statistical tie. Against Vertex, the paired difference is +0.027 with a 95% interval of −0.043 to +0.103. The two rows agree on 52 of 60 questions, and of the eight where they differ, each wins four. Against Qdrant the interval also crosses zero, narrowly. Against Vectorize (+0.111, interval +0.018 to +0.210) and PageIndex (+0.563) the difference is clear. The consistency across judges, set out below, is real, but it is consistency in a small lead. Separating PixelRAG + Jev from Vertex would take a larger or harder question set, and at this ceiling the second matters more than the first.

2. The lead is Jev’s. The ablation row runs the same retriever and the same candidates, interleaved instead of ranked by Jev. It scores 0.817. The paired difference is +0.130 (interval +0.044 to +0.221): twelve questions are better with Jev and two are worse. This is the largest separable effect on the board apart from retrieval itself. Removing one ranking step moves the row from first place to fifth, below Qdrant and Vectorize. The mechanism is visible in retrieval before any answer is written: the tile holding the answer comes first for 46 of 60 questions with Jev and 20 without. Answers scored zero rise from one or two per run to six to nine.

3. The gain comes from the ranking step, not the pixels. Measured component by component before any board run, over the 59 questions whose answer we could locate on a tile, the answer tile came first for 19 questions with vision search alone and 25 with keyword search alone. Either one followed by Jev: 43. Jev over both lists: 51. The image model’s main contribution is finding candidates that keyword search misses. On this corpus it could hardly be otherwise. 983 of its 1,001 files are transcripts, recipes and product pages that exist only as text, which we typeset into PDFs so the image model had pages to look at. A corpus where layout carries meaning, such as tables, figures or scanned forms, would be a fairer test of what PixelRAG itself is for.

4. The same step transfers to a different retriever. PageIndex, an open-source retriever that we host unmodified, navigates a tree of document summaries. With Jev choosing which nodes reach the answering model, it scores 0.8125 against plain PageIndex’s 0.384. The two rows were scored by different judges, which is why the board separates them. Under the same judge, 2.5 Flash, the Jev configuration’s median over four passes is 0.823, so the judge moves the number by about a point and leaves a gap of more than forty. Jev gives two structurally unrelated retrievers large and separable improvements. In PixelRAG it ranks candidate tiles; in PageIndex it chooses which parts of the tree to open.

Robustness to the judge

A row that is best under one judge may only be best under that judge. We re-scored PixelRAG + Jev’s stored answers with the two other judges we use. DeepSeek V4 Flash gives it 0.942 against Vertex’s 0.910, and Gemini 3.8 Flash gives it 0.963 against 0.929. It is first under all three judges, by about three points each time, which is the same small lead the paired analysis finds.

We then asked the obvious harder question: can Jev itself serve as the judge, given that it ranks inside two of the rows it would be judging? We re-scored all 657 stored answers on both retrieval boards offline, with Jev classifying each claim in each reference answer as supported, hedged, contradicted or omitted. Gemini 2.5 Flash made the same classifications from the same fixed list of claims, and both went through the scorer’s own arithmetic, so the judge was the only variable. Nothing was published, and the board did not change.

• Agreement. On the rubric’s wording verbatim, Jev and Gemini agreed on 88.8% of claim verdicts. Almost all the disagreement was one kind of answer: one that declines (“I don’t have reliable information about…”) and then lists generic possibilities. The rubric defines a hedge as an answer that “mentions the topic but softens, waffles, or qualifies”. Jev applied that wording literally and gave such refusals half credit, while Gemini applied the evident intent and gave them none. Jev’s own confidence marked the cases: 0.93 on average where the two judges agreed, 0.53 where they did not.
• With the rubric clarified. Given the same single clarifying sentence (a refusal omits a claim; it does not hedge it), the two agreed on 94.7% of verdicts. That matches Gemini’s agreement with itself across the same change of wording, 94.6%. Among the rows scored by 2.5 Flash, Jev’s ordering matched the published board exactly.
• No self-preference. The pair that differs only by Jev, PixelRAG with and without it, is the one place where a self-interested judge would show. Jev judged the gap between them smaller than Gemini did, in both passes (0.089 against 0.111, and 0.102 against 0.115).
• Throughput. Jev scored all 657 answers in under a minute, with no failed requests.

Two things follow from this. The rubric used on these boards has an ambiguity that a literal reader exposes. Fixing it changes the scorer’s version, so it has to ship with a migration of existing human ratings, and we have recorded it rather than patch it in place. And agreement between judges is not accuracy. No human-rated set exists for these questions yet, and about sixty answers weighted toward the disagreements would calibrate every judge at once. If a Jev-judged run is ever published, the board will give it a table of its own, as it now does for every judge. This re-score was offline and unsigned, so it is reported here rather than on the board.

Threats to validity

It was designed on the test. We chose this architecture (vision, keyword or both; thirty candidates each; Jev ranking the union) by measuring retrieval on these same sixty questions. Those were three coarse options, not a tuning sweep, but none of the original four rows was ever shaped by these questions at all. The dataset section above asks whether an expected answer was derived from a system under test. This is its twin, and it belongs on the same list: was the system under test designed against these items? The remedy is a held-out question set written after the design is frozen. We have not written one yet. It is the caveat we would weigh most heavily, because unlike judge noise it cannot be given a number.

The evidence explains the ranking, except where the disclosure says it will not. For every row we measured whether the sentence holding each answer actually reached the answering model: PixelRAG + Jev 53 of 60, Qdrant 40, Vectorize 33, Vertex 24, PageIndex 7. Answer scores follow that order with one exception. Vertex delivers the exact evidence sentence only 24 times and still scores 0.920. Vertex searches a newer and larger ingestion of the corpus, which the board discloses, parsed and cut differently. So it can deliver the same fact in words our exact-sentence check does not match, or from another passage entirely. The two measurements disagree in exactly the place the disclosed caveat says the rows differ, which is what a disclosure is for.

A component that fails open to absence. The retriever ran on development hardware behind a tunnel. In one early pass, three image searches exceeded production’s five-second limit, so those questions were answered with no retrieval at all, and the pass still reported a plausible 0.893. Nothing in the stack treats an empty retrieval as an error: the model answers from nothing, the judge scores the answer, the run reports complete, and its manifest would have signed and verified. We admitted a pass to the board only if the retriever’s own log showed sixty calls, each inside the time limit. This is a fourth way a score quietly stops meaning anything, and the most misleading one, because the thing it gets wrong is the thing being measured: a stack that is down scores like a stack that is weak. The platform already stores the retrieved context for every test, so a manifest could carry how many turns received any context, and a board could refuse a run in which the stack was missing. Today that check is ours, not the document’s.

Imputed items are not disclosed. In two of the three published runs, one question lacks a score of its own: one errored after its retrieval succeeded, and in the other the judge never returned a score while the run still reported sixty passes. Publication filled each gap with that question’s answer from the previous pass. It is the same rule every row on the board has been through, and it is defensible. What is not defensible is that neither the run nor its signed manifest mentions it.

The name is not in the signature. A retriever that a workspace registers itself is published under an opaque digest, so the signed row reads External retrieval tool (d810847b) + External retrieval tool (1df1e02a). The leaderboard is unauthenticated, and we will not let a workspace write arbitrary text onto it. Our board page shows the row as PixelRAG + Jev because the workspace is ours and this post is its write-up. That name is an annotation our page adds, matched to the exact stack id. The API does not serve it and the manifest does not contain it. It is the a label is not a document problem seen from the other side.

Cost. Jev adds about a tenth of a cent per question. Retrieval takes a median of 1.2 seconds against Qdrant’s 0.5, on development hardware.

What we are not claiming

In the spirit of the thing:

• Our published rows are republished, not measured. The manifests say so and the verifier warns about it.
• Several of our platform benchmarks are too small to rank on — some carry five samples or fewer, two carry exactly one. A one-sample benchmark can only ever score 0.0 or 1.0. They are not on any public board and should not be.
• Two of our earlier boards saturate: one puts twelve of sixteen models at exactly 1.0000. Twelve models “tied for first” teaches a reader nothing. That is a content problem, not a harness problem, and it is fixed by writing harder samples.
• We cannot currently run the closed frontier models as baselines. The harness has clients for three backends and those four are not among them.
• Both retrieval boards use one answering model. Three judges agree on the ordering, and Jev agrees once the rubric’s definition of a hedge is clarified (see the footnote above and the update), but none is calibrated against a human rater, so accuracy is unestablished. On the nutrition board, the Vertex row searches a newer and larger ingestion than the others, and the PageIndex + Jev row was scored by Gemini 3.8 Flash, so the board ranks it in a separate table.
• PixelRAG + Jev was designed on the same sixty questions it is scored on. Its lead over Vertex is consistent across three judges, but on paired questions it is not statistically separable from Vertex or Qdrant. See the update.

Since 25 September, each board also has a full test specification: what it tests, how every answer is scored, and each known defect, including two worse than anything above. The top row of Red Team Core rested partly on empty answers, since fixed in a version 1.1.0 of both red-team boards with every model re-run; and the erasure board’s single number cannot show whether the erasure worked.

None of that is fatal and all of it is written down. A benchmark that cannot state its own limitations is asking for the same trust it exists to replace.

Where this goes

The order is: harness attestation, so a score can say the sandbox was not gamed. Then dataset provenance, so a benchmark can say who wrote it and where its answers came from. Then model provenance, so a result can say which weights produced it. Then the manifest format goes to the standards bodies, because a verification format owned by one company is just a second thing to trust.

Attestation says this score is real. It does not say this model is safe. Keeping those two apart is most of the work, and conflating them is how this entire category fails.

If you want to check any of it, nothing here requires an account. The leaderboards are public, the verifier is on npm, and every row links to the manifest that contains its score. Fetch the manifest and its outputs, run verify(), and read the warnings — including ours.

Related: Calibrating the Judge on anchoring an LLM grader to a human rater · Inside the RAG Arena on where these scored-QA runs come from · Deleting Paris from a Language Model on receipts for knowledge editing, which use the same attestation machinery.