rewirebio.iobenchmarks
Evidence and investigation

Contested results

Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.

These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.

A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.

How the check works, and how each follow-up is recorded: discrepancies between studies.

In this release

  • 121open
  • 5settled
  • 7resolved
  • 1medium severity
  • 132low severity

Release 2026-11-02-a7f03c9e55a5 · 133 findings in all · rules version 4

Findings

Status: All · Open · Settled · Resolved

Severity: All · Medium · Low

State: All · Not yet reviewed · Under investigation · Explained · Confirmed error · Not a real disagreement

Showing findings that involve results in Rank rare-disease variants for review, from the study that disagrees or from the agreeing group. All use cases

1 finding matches, most severe first.

ESM-1b and EVE: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Under investigation
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks ESM-1b above EVE in 6 of 6 comparisons; 2 of 2 other studies rank EVE above ESM-1b.

2 papers rank EVE above ESM-1b in 22 of their 26 comparisons and ESM-1b higher in 1; 3 are ties. Brandes et al. 2023 ranks ESM-1b above EVE in all 6 comparisons that are not ties (2 are ties).

Comparisons in which each paper scores EVE or ESM-1b higher2 other papers: EVE higher in 22, ESM-1b higher in 1, 3 ties. Brandes et al. 2023 (disagrees): EVE higher in 0, ESM-1b higher in 6, 2 ties.2 other papers22 EVE higher223 tie31 ESM-1b higherDisagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods2 tie26 ESM-1b higher6
EVE higherTieESM-1b higher
Compared
ESM-1b, EVE

1 follow-up action, latest 23 October 2026. Next check 6 November 2026