Contested result

Evo and AIDO.RNA: RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks AIDO.RNA above Evo in 4 of 4 comparisons; other studies rank Evo above AIDO.RNA in 5 of 5.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Zero-shot benchmarking of RNA language… ranks Evo above AIDO.RNA in all 5 comparisons that are not ties (1 is a tie). RNAGym fitness leaderboard ranks AIDO.RNA above Evo in all 4 of its comparisons.

Rank Evo higher (1): Zero-shot benchmarking of RNA language…
Rank AIDO.RNA higher (1): RNAGym fitness leaderboard (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Evo higherTieAIDO.RNA higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks Evo higher	Ranks AIDO.RNA higher	Ties
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md · Repository	Disagrees	0	4	0
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOI	Agrees	5	0	1

Comparisons in RNAGym fitness leaderboard (4)

Protocol	Dataset	Metric	Evo	AIDO.RNA	Higher
RNAGym v0.1.1 fitness, signed Spearman, Aptamer	RNAGym Aptamer DMS assays (2)	Spearman (mean signed Spearman over the 2 Aptamer assays), unitless	0.0058	0.0939	AIDO.RNA
RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categories	RNAGym ncRNA DMS assays (31)	Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless	0.138	0.182	AIDO.RNA
RNAGym v0.1.1 fitness, signed Spearman, Ribozyme	RNAGym Ribozyme DMS assays (26)	Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless	0.0278	0.074	AIDO.RNA
RNAGym v0.1.1 fitness, signed Spearman, tRNA	RNAGym tRNA DMS assays (3)	Spearman (mean signed Spearman over the 3 tRNA assays), unitless	0.385	0.393	AIDO.RNA

Comparisons in Zero-shot benchmarking of RNA language… (6)

Protocol	Dataset	Metric	Evo	AIDO.RNA	Higher
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	AUROC (PLL-D; mean over assays), unitless	0.588	0.585	Tie
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	AUROC (PLL-D; median over assays), unitless	0.586	0.564	Evo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	MCC (PLL-D; mean over assays), unitless	0.132	0.122	Evo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	MCC (PLL-D; median over assays), unitless	0.13	0.098	Evo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	Absolute spearman correlation (PLL-D; mean over assays), unitless	0.179	0.175	Evo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	Absolute spearman correlation (PLL-D; median over assays), unitless	0.178	0.136	Evo

What disagrees

Other studies: rank Evo above AIDO.RNA in 5 of 5 comparisons
This study: ranks AIDO.RNA above Evo in 4 of 4 comparisons

Compared: Evo, AIDO.RNA
Study that disagrees: RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md
Other studies: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning, Zero-shot benchmarking of RNA language models: Supplementary Information
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (8)
Study	Tool	Protocol or benchmark	Metric	Value
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	AIDO.RNA (1.6B) (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categories	Spearman (macro-mean over 3 ncRNA categories (equal category weights))	0.182
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	AIDO.RNA (1.6B) (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, Ribozyme	Spearman (mean signed Spearman over the 26 Ribozyme assays)	0.074
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	AIDO.RNA-650M (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, Aptamer	Spearman (mean signed Spearman over the 2 Aptamer assays)	0.0939
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	AIDO.RNA-650M (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, tRNA	Spearman (mean signed Spearman over the 3 tRNA assays)	0.393
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	Evo 1.5 (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categories	Spearman (macro-mean over 3 ncRNA categories (equal category weights))	0.138
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	Evo 1.5 (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, Ribozyme	Spearman (mean signed Spearman over the 26 Ribozyme assays)	0.0278
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	Evo 1.5 (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, tRNA	Spearman (mean signed Spearman over the 3 tRNA assays)	0.385
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md	Evo 1 (RNAGym fitness leaderboard v0.1.1)	RNAGym v0.1.1 fitness, signed Spearman, Aptamer	Spearman (mean signed Spearman over the 2 Aptamer assays)	0.0058

In the other studies (10)
Study	Tool	Protocol or benchmark	Metric	Value
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	AIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)	AUROC (PLL-D; median over assays)	0.564
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	AIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)	MCC (PLL-D; median over assays)	0.098
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	AIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)	Absolute spearman correlation (PLL-D; mean over assays)	0.175±0.135
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	AIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)	Absolute spearman correlation (PLL-D; median over assays)	0.136
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	AIDO.RNA (650M) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)	MCC (PLL-D; mean over assays)	0.122±0.107
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)	AUROC (PLL-D; median over assays)	0.586
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)	MCC (PLL-D; mean over assays)	0.132±0.088
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)	MCC (PLL-D; median over assays)	0.13
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)	Absolute spearman correlation (PLL-D; mean over assays)	0.179±0.114
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	EVO1.5 (8k_base) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)	Absolute spearman correlation (PLL-D; median over assays)	0.178

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-15227390e85414ca · All contested results · How findings are made and followed up