Contested result

RNAErnie and Evo: RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above Evo in 4 of 4 comparisons; other studies rank Evo above RNAErnie in 6 of 6.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Zero-shot benchmarking of RNA language… ranks Evo above RNAErnie in all 6 of its comparisons. RNAGym fitness leaderboard ranks RNAErnie above Evo in all 4 of its comparisons.

Rank Evo higher (1): Zero-shot benchmarking of RNA language…
Rank RNAErnie higher (1): RNAGym fitness leaderboard (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Evo higherTieRNAErnie higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks Evo higher	Ranks RNAErnie higher	Ties
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md · Repository	Disagrees	0	4	0
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOI	Agrees	6	0	0

Comparisons in RNAGym fitness leaderboard (4)

Protocol	Dataset	Metric	Evo	RNAErnie	Higher
RNAGym v0.1.1 fitness, signed Spearman, Aptamer	RNAGym Aptamer DMS assays (2)	Spearman (mean signed Spearman over the 2 Aptamer assays), unitless	0.0058	0.0306	RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categories	RNAGym ncRNA DMS assays (31)	Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless	0.138	0.194	RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, Ribozyme	RNAGym Ribozyme DMS assays (26)	Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless	0.0278	0.134	RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, tRNA	RNAGym tRNA DMS assays (3)	Spearman (mean signed Spearman over the 3 tRNA assays), unitless	0.385	0.416	RNAErnie

Comparisons in Zero-shot benchmarking of RNA language… (6)

Protocol	Dataset	Metric	Evo	RNAErnie	Higher
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	AUROC (PLL-D; mean over assays), unitless	0.588	0.551	Evo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	AUROC (PLL-D; median over assays), unitless	0.586	0.541	Evo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	MCC (PLL-D; mean over assays), unitless	0.132	0.076	Evo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	MCC (PLL-D; median over assays), unitless	0.13	0.051	Evo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	Absolute spearman correlation (PLL-D; mean over assays), unitless	0.179	0.1	Evo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	Absolute spearman correlation (PLL-D; median over assays), unitless	0.178	0.081	Evo

What disagrees

Other studies: rank Evo above RNAErnie in 6 of 6 comparisons
This study: ranks RNAErnie above Evo in 4 of 4 comparisons

Compared: RNAErnie, Evo
Study that disagrees: RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md
Other studies: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning, Zero-shot benchmarking of RNA language models: Supplementary Information
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (8)
Study	Tool	Protocol or benchmark	Metric	Value
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	Evo 1.5 (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categories	Spearman (macro-mean over 3 ncRNA categories (equal category weights))	0.138
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	Evo 1.5 (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, Ribozyme	Spearman (mean signed Spearman over the 26 Ribozyme assays)	0.0278
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	Evo 1.5 (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, tRNA	Spearman (mean signed Spearman over the 3 tRNA assays)	0.385
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	Evo 1 (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, Aptamer	Spearman (mean signed Spearman over the 2 Aptamer assays)	0.0058
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	RNA-ERNIE (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, Aptamer	Spearman (mean signed Spearman over the 2 Aptamer assays)	0.0306
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	RNA-ERNIE (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categories	Spearman (macro-mean over 3 ncRNA categories (equal category weights))	0.194
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	RNA-ERNIE (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, Ribozyme	Spearman (mean signed Spearman over the 26 Ribozyme assays)	0.134
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	RNA-ERNIE (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, tRNA	Spearman (mean signed Spearman over the 3 tRNA assays)	0.416

In the other studies (12)
Study	Tool	Protocol or benchmark	Metric	Value
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)	AUROC (PLL-D; mean over assays)	0.588±0.056
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)	AUROC (PLL-D; median over assays)	0.586
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)	MCC (PLL-D; mean over assays)	0.132±0.088
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)	MCC (PLL-D; median over assays)	0.13
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)	Absolute spearman correlation (PLL-D; mean over assays)	0.179±0.114
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)	Absolute spearman correlation (PLL-D; median over assays)	0.178
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNAErnie (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)	AUROC (PLL-D; mean over assays)	0.551±0.045
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNAErnie (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)	AUROC (PLL-D; median over assays)	0.541
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNAErnie (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)	MCC (PLL-D; mean over assays)	0.076±0.069
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNAErnie (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)	MCC (PLL-D; median over assays)	0.051
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNAErnie (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)	Absolute spearman correlation (PLL-D; mean over assays)	0.100±0.092
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNAErnie (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)	Absolute spearman correlation (PLL-D; median over assays)	0.081

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-1c82736bf8936ab1 · All contested results · How findings are made and followed up