rewirebio.iobenchmarks
Evidence and investigation

Contested results

Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.

These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.

A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.

How the check works, and how each follow-up is recorded: discrepancies between studies.

In this release

  • 192open
  • 6settled
  • 7resolved
  • 3medium severity
  • 202low severity

Release 2026-11-21-c8bea25e5abb · 205 findings in all · rules version 5

Findings

Status: All · Open · Settled · Resolved

Severity: All · Medium · Low

State: All · Not yet reviewed · Under investigation · Explained · Confirmed error · Not a real disagreement

Showing findings that compare SpliceAI. All models

1 finding matches, most severe first.

Pangolin and SpliceAI: Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) ranks SpliceAI above Pangolin in 27 of 28 comparisons; 3 of 3 other studies rank Pangolin above SpliceAI.

7 papers rank Pangolin above SpliceAI in 29 of their 72 comparisons and SpliceAI higher in 18; 25 are ties. Genome modelling and design across all… ranks SpliceAI above Pangolin in 27 of its 28 comparisons and Pangolin higher in 1.

Comparisons in which each paper scores Pangolin or SpliceAI higher7 other papers: Pangolin higher in 29, SpliceAI higher in 18, 25 ties. Genome modelling and design across all… (disagrees): Pangolin higher in 1, SpliceAI higher in 27, 0 ties.7 other papers29 Pangolin higher2925 tie2518 SpliceAI higher18Disagrees: Genome modelling and design across all…Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026)1 Pangolin higher27 SpliceAI higher27
Pangolin higherTieSpliceAI higher
Compared
Pangolin, SpliceAI

Not reviewed yet.