rewirebio.iobenchmarks

PolicyRelease 2026-11-21-adc15377fab2

Independence policy

Who runs this database, how placement on a board is decided, how each result is labelled by who produced it, and what happens when someone says a result is wrong.

Who runs it

The database is part of Rewire, a publication and computational project written and maintained by Tim Richardson. The project is described on rewirebio.io.

The data is not sold. Rewire does not sell the benchmark data or the graph, license them to a partner or put them behind a paywall, in whole or in part. The JSON API needs no key and the release files are free to download. Terms of reuse are on the data licence page.

How placement is decided

The order of rows within a board is not chosen by hand. Each board ranks one use case on one protocol and one metric, in the order of the recorded values, and every row comes from a result record in the release. There is no composite score across protocols. The rules that pick the protocol and build the rows are in the site's public code, and the records are in the data repository.

A board marked Proxy ranks on a protocol that was judged a proxy for the use case, not a direct test of it. It is used only when the use case has no directly relevant protocol that can be ranked, and it keeps the label wherever it is shown. In release 2026-11-21-adc15377fab2, 19 boards are proxy, out of 30 boards that rank anything.

Which boards the homepage shows, and in what order, is chosen by hand. The choice is written in the site's code and works like this. A board can appear only if it ranks at least 10 distinct tools; a proxy board must also have at least 5 foundation-model rows and more than half its rows from independent studies. Directly relevant boards come first. Within that, the order is Rewire's. Every use case has its board on its own page under use cases, whether or not the homepage shows it.

The homepage shows these boards, in this order:

These boards meet the rule and are left off the homepage by hand, each for the reason given:

A model's developers have the same route as anyone else to add evidence or ask for a correction.

How results are labelled by who produced them

Each evaluation in the release carries an origin, and each ranked row and result page prints it. The origin is one of these:

Developer-reported
Reported by the developers of the evaluated tool.
Independent study
Reported in a study whose authors did not develop the evaluated tool.
Rewire run
Run by Rewire, not transcribed from a publication. Rewire wrote some of the baselines and controls it runs, so a Rewire run is not counted as independent of the tool's developers.
Quoted from another paper
A value the source copied from another paper or benchmark table.
Source does not say who ran it
The source prints the value but does not say who ran the tool, so it is not counted as independent.

The label is only as good as the source. Of 771 ranked rows across all boards in release 2026-11-21-adc15377fab2, 99 read "Source does not say who ran it" and 0 read "Origin not recorded".

Only an independent study counts as independent of a tool's developers. A developer-reported value is shown and ranked like any other, with its label beside it. A value whose source does not say who ran the tool is not counted as independent.

How Rewire's own results are produced and receipted

A result that Rewire ran itself is labelled Rewire run. It is not counted as independent. Rewire scores some methods that it wrote: baselines and controls in its own runs. Those results carry the Rewire run label like the rest. The runner is published in the benchmark runner repository.

Checking a value against its source is not the same as rerunning it. Source-checked means the recorded value was compared with the source it cites. On a leaderboard and in the JSON API, a result reads Reproduced only when the result or its evaluation cites a run receipt: the record of an executed run of the full protocol, held in the release. A record's own status in the release can say that a rerun was recorded without a receipt being cited. On this site's pages that status reads Rerun recorded, not Reproduced; the release files keep the status as the data repository wrote it.

Release 2026-11-21-adc15377fab2 holds 1 run receipt. No ranked row cites one, so no row on any board reads Reproduced.

Review of the records is automated. No statement on this site has had human sign-off unless the page says so. See evidence and sources for what each review status means.

Corrections and disputes

To report a wrong value, a wrong attribution or a missing result, use any of these:

Nothing sent in is published automatically. A correction is made by changing the record in the data repository and publishing a new release. Earlier releases are not rewritten, and a record that has been replaced stays in the release, marked superseded.

When studies disagree, the site does not pick a side by hand. Each release runs an automatic check over the studies that report on the same models and lists what it finds under contested results, with the studies on each side. Release 2026-11-21-adc15377fab2 lists 205 findings, by the rule that raised each:

Two-study conflict: 124
Exactly two studies compare a pair of models and they disagree. Neither is named as the one that is wrong.
Ordering reversal: 73
At least two other studies agree on which of two models is better, and one study finds the opposite.
Resolved, rule not recorded: 7
A settled finding that the check no longer raises. It stays listed so that its follow-up can be read.
Gap outlier: 1
One study's gap between a model and a shared reference model lies well outside the gaps the other studies report.

A finding is a lead to check, not a finding of error. A finding by itself never hides a result: the results it names stay in every table, page and download, marked Contested. A result is withheld only when it is marked disputed. The follow-up on each finding is recorded with it and shown: what was investigated, what was reported to the authors and what was corrected. The rules are in the discrepancy documentation.

If you developed a tool and think a result misrepresents it, the routes above are open to you. A reply from a tool's developers can be recorded against the finding it answers, and a recorded reply is shown with that finding. Not every reply received so far has been recorded. A reply does not remove the other studies' results.

Takedown

A rights holder who wants a stored copy of a source removed should write to tim@rewire.it. The copy is removed from the release assets, the private store and the working caches, and is not fetched again; the citation stays. One limit applies: a file that was committed to the public data repository cannot be removed from that repository's git history. Which sources that affects, and the rest of the detail, are on the data licence page.