rewirebio.iobenchmarks
Contested result

RNAErnie and AIDO.RNA: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md disagree

  • Low severity
  • Explained
  • two-study-conflict
  • Same task, different protocols

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks AIDO.RNA above RNAErnie in 36 of 36 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above AIDO.RNA in 3 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank RNAErnie above AIDO.RNA in 3 of their 40 comparisons and AIDO.RNA higher in 37. Of those papers, Zero-shot benchmarking of RNA language… ranks AIDO.RNA higher on its own (36 of 36).

Papers grouped by which system they rank higher: RNAErnie or AIDO.RNARank RNAErnie higher: RNAGym fitness leaderboard. Rank AIDO.RNA higher: Zero-shot benchmarking of RNA language…. Mixed: none.Rank RNAErnie higherMixedRank AIDO.RNA higherRNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdZero-shot benchmarkingof RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning
  • Rank RNAErnie higher (1): RNAGym fitness leaderboard
  • Rank AIDO.RNA higher (1): Zero-shot benchmarking of RNA language…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores RNAErnie or AIDO.RNA higherZero-shot benchmarking of RNA language…: RNAErnie higher in 0, AIDO.RNA higher in 36, 0 ties. RNAGym fitness leaderboard: RNAErnie higher in 3, AIDO.RNA higher in 1, 0 ties.Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning36 AIDO.RNA higher36RNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md3 RNAErnie higher31 AIDO.RNA higher1
RNAErnie higherTieAIDO.RNA higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks RNAErnie higherRanks AIDO.RNA higherTies
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOIAgrees0360
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md · RepositoryAgrees310
Comparisons in Zero-shot benchmarking of RNA language… (36)
ProtocolDatasetMetricRNAErnieAIDO.RNAHigher
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (WT-LLR; mean over assays), unitless0.5810.617AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (WT-LLR; median over assays), unitless0.5680.595AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (WT-LLR; mean over assays), unitless0.1250.171AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (WT-LLR; median over assays), unitless0.0990.155AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (WT-LLR; mean over assays), unitless0.1620.237AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (WT-LLR; median over assays), unitless0.1330.214AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; mean over assays), unitless0.5510.585AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; median over assays), unitless0.5410.564AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; mean over assays), unitless0.0760.121AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; median over assays), unitless0.0510.098AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; mean over assays), unitless0.10.175AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; median over assays), unitless0.0810.136AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.2880.508AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless0.2380.579AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless0.290.518AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless0.2360.595AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.240.512AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless0.2310.6AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless0.2430.52AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless0.2370.618AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.7090.836AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.8540.931AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.7750.859AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless0.5320.717AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.5720.718AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.7650.805AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.7130.767AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless0.3670.532AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.7510.868AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.880.949AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.8010.884AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless0.5940.767AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.6120.756AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.7840.891AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.7280.802AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless0.4040.591AIDO.RNA
Comparisons in RNAGym fitness leaderboard (4)
ProtocolDatasetMetricRNAErnieAIDO.RNAHigher
RNAGym v0.1.1 fitness, signed Spearman, AptamerRNAGym Aptamer DMS assays (2)Spearman (mean signed Spearman over the 2 Aptamer assays), unitless0.03060.0847AIDO.RNA
RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesRNAGym ncRNA DMS assays (31)Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless0.1940.182RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, RibozymeRNAGym Ribozyme DMS assays (26)Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless0.1340.074RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, tRNARNAGym tRNA DMS assays (3)Spearman (mean signed Spearman over the 3 tRNA assays), unitless0.4160.389RNAErnie

What disagrees

studies
2
Compared
RNAErnie, AIDO.RNA
Study that disagrees
Other studies
Rule
two-study-conflict; same task, different protocols
Model identity
Matched across studies partly through links not yet reviewed
Earlier IDs
discrepancy-8c69346a6089882b

Follow-up

Current state: Explained. Settled.

  1. 23 October 2026: Investigated · rewire-bio/rewire-benchmark-data#209

    State after this step: Under investigation

    Both sides score the RNAGym ncRNA DMS assays. The leaderboard (signed Spearman) ranks RNAErnie first (macro 0.194 against 0.182); Wang 2026 (absolute Spearman per assay, PLL-D and WT-LLR scoring) ranks AIDO.RNA first (0.175 against 0.100, 0.237 against 0.162). Working hypothesis: the absolute value and the scoring rule, not an error. Being checked against both sources' methods (issue 209).

    Next check: 6 November 2026

  2. 23 October 2026: Investigated · rewire-bio/rewire-benchmark-data#209

    State after this step: Explained

    Both sides score the same 31 RNAGym ncRNA assays (26 ribozyme, 3 tRNA, 2 aptamer, all at most 425 nt, so Wang's 440 nt filter keeps every one). The order differs because the v0.1.1 leaderboard's RNAErnie score is not comparable with its AIDO.RNA score. The RNAGym maintainers later stated that RNAErnie's scorer takes one pass over the unmasked wild type, a softmax rather than a log-softmax, and sums raw probability differences at the mutated positions, so its 0.1937 is 'not comparable with the masked checkpoints' (commit d43db8c613); it was carried over unverified because it needs a paddlepaddle environment. At v0.1.1 AIDO.RNA was scored with mut-fill, the weakest of four fill strategies; rescored with wt-fill it 'takes second place from RNA-ERNIE' (commit d024cd81a3). The v0.2 leaderboard ranks AIDO.RNA 650M (0.21) and 1.6B (0.20) above RNA-ERNIE (0.19), the order Wang reports. RNA-ERNIE's ribozyme lead (0.14) falls to about 0.02 within a mutation count, and mutation count alone scores 0.20 on ribozymes, so its lead mostly reflects how many mutations a variant has. Wang's SR is the absolute Spearman (Methods), which RNAGym's own script also reported until commit ac371bff28; its effect on this pair cannot be measured without per-assay predictions, and it is not needed to explain the reversal.

  3. 24 October 2026: Explained · rewire-bio/rewire-benchmark-data#209

    State after this step: Explained

    Both sides score the same 31 RNAGym ncRNA assays (26 ribozyme, 3 tRNA, 2 aptamer, all at most 425 nt, so Wang's 440 nt filter keeps every one). The order differs because the v0.1.1 leaderboard's RNAErnie score is not comparable with its AIDO.RNA score. The RNAGym maintainers later stated that RNAErnie's scorer takes one pass over the unmasked wild type, a softmax rather than a log-softmax, and sums raw probability differences at the mutated positions, so its 0.1937 is 'not comparable with the masked checkpoints' (commit d43db8c613); it was carried over unverified because it needs a paddlepaddle environment. At v0.1.1 AIDO.RNA was scored with mut-fill, the weakest of four fill strategies; rescored with wt-fill it 'takes second place from RNA-ERNIE' (commit d024cd81a3). The v0.2 leaderboard ranks AIDO.RNA 650M (0.21) and 1.6B (0.20) above RNA-ERNIE (0.19), the order Wang reports. RNA-ERNIE's ribozyme lead (0.14) falls to about 0.02 within a mutation count, and mutation count alone scores 0.20 on ribozymes, so its lead mostly reflects how many mutations a variant has. Wang's SR is the absolute Spearman (Methods), which RNAGym's own script also reported until commit ac371bff28; its effect on this pair cannot be measured without per-assay predictions, and it is not needed to explain the reversal.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-055eb0dd61040574 · All contested results · How findings are made and followed up