rewirebio.iobenchmarks
Evidence and investigation

Contested results

Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.

These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.

A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.

How the check works, and how each follow-up is recorded: discrepancies between studies.

In this release

  • 192open
  • 6settled
  • 7resolved
  • 3medium severity
  • 202low severity

Release 2026-11-21-c8bea25e5abb · 205 findings in all · rules version 5

Findings

Status: All · Open · Settled · Resolved

Severity: All · Medium · Low

State: All · Not yet reviewed · Under investigation · Explained · Confirmed error · Not a real disagreement

Showing findings that compare Evo. All models

6 findings match, most severe first.

Evo 2 and Evo: BacBench LICENSE file at commit 64b7d45 reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

BacBench LICENSE file at commit 64b7d45 ranks Evo above Evo 2 in 8 of 10 comparisons; 2 of 2 other studies rank Evo 2 above Evo.

2 papers rank Evo 2 above Evo in all 94 of their comparisons. BacBench LICENSE file at commit 64b7d45 ranks Evo above Evo 2 in 8 of its 10 comparisons and Evo 2 higher in 2.

Comparisons in which each paper scores Evo 2 or Evo higher2 other papers: Evo 2 higher in 94, Evo higher in 0, 0 ties. BacBench LICENSE file at commit 64b7d45 (disagrees): Evo 2 higher in 2, Evo higher in 8, 0 ties.2 other papers94 Evo 2 higher94Disagrees: BacBench LICENSE file at commit 64b7d45BacBench LICENSE file at commit 64b7d452 Evo 2 higher28 Evo higher8
Evo 2 higherTieEvo higher
Compared
Evo 2, Evo

Not reviewed yet.

Nucleotide Transformer and Evo: RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks Evo above Nucleotide Transformer in 3 of 4 comparisons; 2 of 2 other studies rank Nucleotide Transformer above Evo.

3 papers rank Nucleotide Transformer above Evo in 72 of their 145 comparisons and Evo higher in 65; 8 are ties. Of those papers, Genome modelling and design across all… ranks Evo higher on its own (61 of 90). RNAGym fitness leaderboard ranks Evo above Nucleotide Transformer in 3 of its 4 comparisons and Nucleotide Transformer higher in 1.

Comparisons in which each paper scores Nucleotide Transformer or Evo higher3 other papers: Nucleotide Transformer higher in 72, Evo higher in 65, 8 ties. RNAGym fitness leaderboard (disagrees): Nucleotide Transformer higher in 1, Evo higher in 3, 0 ties.3 other papers72 Nucleotide Transformer higher728 tie65 Evo higher65Disagrees: RNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md1 Nucleotide Transformer higher13 Evo higher3
Nucleotide Transformer higherTieEvo higher
Compared
Nucleotide Transformer, Evo

Not reviewed yet.

DNABERT-2 and Evo: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above DNABERT-2 in 6 of 6 comparisons; 2 of 2 other studies rank DNABERT-2 above Evo.

2 papers rank DNABERT-2 above Evo in 45 of their 55 comparisons and Evo higher in 10. Zero-shot benchmarking of RNA language… ranks Evo above DNABERT-2 in all 6 of its comparisons.

Comparisons in which each paper scores DNABERT-2 or Evo higher2 other papers: DNABERT-2 higher in 45, Evo higher in 10, 0 ties. Zero-shot benchmarking of RNA language… (disagrees): DNABERT-2 higher in 0, Evo higher in 6, 0 ties.2 other papers45 DNABERT-2 higher4510 Evo higher10Disagrees: Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning6 Evo higher6
DNABERT-2 higherTieEvo higher
Compared
DNABERT-2, Evo

Not reviewed yet.

Evo and LucaOne: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and GENEB leaderboard (leaderboard.json, 13 task-type aggregates) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above LucaOne in 6 of 6 comparisons; GENEB leaderboard (leaderboard.json, 13 task-type aggregates) ranks LucaOne above Evo in 40 of 45 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank Evo above LucaOne in 11 of their 51 comparisons and LucaOne higher in 40. Of those papers, GENEB leaderboard ranks LucaOne higher on its own (40 of 45).

Comparisons in which each paper scores Evo or LucaOne higher2 other papers: Evo higher in 11, LucaOne higher in 40, 0 ties.2 other papers11 Evo higher1140 LucaOne higher40
Evo higherTieLucaOne higher
Compared
Evo, LucaOne

Not reviewed yet.

RNAErnie and Evo: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • Same task, different protocols

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above RNAErnie in 6 of 6 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above Evo in 4 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank RNAErnie above Evo in 4 of their 10 comparisons and Evo higher in 6. Of those papers, Zero-shot benchmarking of RNA language… ranks Evo higher on its own (6 of 6).

Comparisons in which each paper scores RNAErnie or Evo higher2 other papers: RNAErnie higher in 4, Evo higher in 6, 0 ties.2 other papers4 RNAErnie higher46 Evo higher6
RNAErnie higherTieEvo higher
Compared
RNAErnie, Evo

Not reviewed yet.

Evo and AIDO.RNA: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • Same task, different protocols

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above AIDO.RNA in 5 of 5 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks AIDO.RNA above Evo in 3 of 3 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank Evo above AIDO.RNA in 5 of their 10 comparisons and AIDO.RNA higher in 3; 2 are ties. Of those papers, RNAGym fitness leaderboard ranks AIDO.RNA higher on its own (3 of 4).

Comparisons in which each paper scores Evo or AIDO.RNA higher2 other papers: Evo higher in 5, AIDO.RNA higher in 3, 2 ties.2 other papers5 Evo higher52 tie23 AIDO.RNA higher3
Evo higherTieAIDO.RNA higher
Compared
Evo, AIDO.RNA

Not reviewed yet.