rewirebio.iobenchmarks
Contested result

RNAErnie and AIDO.RNA: RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md reverses the order

  • Medium severity
  • Not yet reviewed
  • Ordering reversal
  • Different tasks across studies

RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above AIDO.RNA in 3 of 4 comparisons; other studies rank AIDO.RNA above RNAErnie in 36 of 36.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Zero-shot benchmarking of RNA language… ranks AIDO.RNA above RNAErnie in all 36 of its comparisons. RNAGym fitness leaderboard ranks RNAErnie above AIDO.RNA in 3 of its 4 comparisons and AIDO.RNA higher in 1.

Papers grouped by which system they rank higher: AIDO.RNA or RNAErnieRank AIDO.RNA higher: Zero-shot benchmarking of RNA language…. Rank RNAErnie higher: RNAGym fitness leaderboard (disagrees). Mixed: none.Rank AIDO.RNA higherMixedRank RNAErnie higherZero-shot benchmarkingof RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md
  • Rank AIDO.RNA higher (1): Zero-shot benchmarking of RNA language…
  • Rank RNAErnie higher (1): RNAGym fitness leaderboard (disagrees)
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores AIDO.RNA or RNAErnie higherRNAGym fitness leaderboard (disagrees): AIDO.RNA higher in 1, RNAErnie higher in 3, 0 ties. Zero-shot benchmarking of RNA language…: AIDO.RNA higher in 36, RNAErnie higher in 0, 0 ties.Disagrees: RNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md1 AIDO.RNA higher13 RNAErnie higher3Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning36 AIDO.RNA higher36
AIDO.RNA higherTieRNAErnie higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks AIDO.RNA higherRanks RNAErnie higherTies
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md · RepositoryDisagrees130
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOIAgrees3600
Comparisons in RNAGym fitness leaderboard (4)
ProtocolDatasetMetricAIDO.RNARNAErnieHigher
RNAGym v0.1.1 fitness, signed Spearman, AptamerRNAGym Aptamer DMS assays (2)Spearman (mean signed Spearman over the 2 Aptamer assays), unitless0.09390.0306AIDO.RNA
RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesRNAGym ncRNA DMS assays (31)Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless0.1820.194RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, RibozymeRNAGym Ribozyme DMS assays (26)Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless0.0740.134RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, tRNARNAGym tRNA DMS assays (3)Spearman (mean signed Spearman over the 3 tRNA assays), unitless0.3930.416RNAErnie
Comparisons in Zero-shot benchmarking of RNA language… (36)
ProtocolDatasetMetricAIDO.RNARNAErnieHigher
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (WT-LLR; mean over assays), unitless0.6170.581AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (WT-LLR; median over assays), unitless0.5950.568AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (WT-LLR; mean over assays), unitless0.1710.125AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (WT-LLR; median over assays), unitless0.1550.099AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (WT-LLR; mean over assays), unitless0.2370.162AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (WT-LLR; median over assays), unitless0.2140.133AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; mean over assays), unitless0.5850.551AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; median over assays), unitless0.5640.541AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; mean over assays), unitless0.1220.076AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; median over assays), unitless0.0980.051AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; mean over assays), unitless0.1750.1AIDO.RNA
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; median over assays), unitless0.1360.081AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.5080.288AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless0.5790.238AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless0.5180.29AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless0.5950.236AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.5120.24AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless0.60.231AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless0.520.243AIDO.RNA
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless0.6180.237AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.8540.709AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.9440.854AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.8730.775AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless0.7450.532AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.7280.572AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.840.765AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.7820.713AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless0.5550.367AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.8890.751AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.9640.88AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.9010.801AIDO.RNA
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless0.8020.594AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.7670.612AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.8970.784AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.810.728AIDO.RNA
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless0.6150.404AIDO.RNA

What disagrees

Other studies
rank AIDO.RNA above RNAErnie in 36 of 36 comparisons
This study
ranks RNAErnie above AIDO.RNA in 3 of 4 comparisons
Compared
RNAErnie, AIDO.RNA
Study that disagrees
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md
Other studies
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning, Zero-shot benchmarking of RNA language models: Supplementary Information
Rule
Ordering reversal; different tasks across studies
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (8)
StudyToolProtocol or benchmarkMetricValue
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdAIDO.RNA (1.6B) (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesSpearman (macro-mean over 3 ncRNA categories (equal category weights))0.182
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdAIDO.RNA (1.6B) (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, RibozymeSpearman (mean signed Spearman over the 26 Ribozyme assays)0.074
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdAIDO.RNA-650M (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, AptamerSpearman (mean signed Spearman over the 2 Aptamer assays)0.0939
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdAIDO.RNA-650M (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, tRNASpearman (mean signed Spearman over the 3 tRNA assays)0.393
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRNA-ERNIE (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, AptamerSpearman (mean signed Spearman over the 2 Aptamer assays)0.0306
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRNA-ERNIE (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesSpearman (macro-mean over 3 ncRNA categories (equal category weights))0.194
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRNA-ERNIE (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, RibozymeSpearman (mean signed Spearman over the 26 Ribozyme assays)0.134
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRNA-ERNIE (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, tRNASpearman (mean signed Spearman over the 3 tRNA assays)0.416
In the other studies (72 results)
In the other studies
StudyToolProtocol or benchmarkMetricValue
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (mean over assays)AUROC (WT-LLR; mean over assays)0.617±0.079
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (median over assays)AUROC (WT-LLR; median over assays)0.595
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (mean over assays)MCC (WT-LLR; mean over assays)0.171±0.115
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (median over assays)MCC (WT-LLR; median over assays)0.155
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (mean over assays)Absolute spearman correlation (WT-LLR; mean over assays)0.237±0.166
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (median over assays)Absolute spearman correlation (WT-LLR; median over assays)0.214
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)AUROC (PLL-D; mean over assays)0.585±0.067
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)AUROC (PLL-D; median over assays)0.564
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)MCC (PLL-D; median over assays)0.098
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)Absolute spearman correlation (PLL-D; mean over assays)0.175±0.135
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)Absolute spearman correlation (PLL-D; median over assays)0.136
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)F1 (attention; mean over structures)0.508±0.263
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)F1 (attention; median over structures)0.579
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)MCC (attention; mean over structures)0.518±0.258
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)MCC (attention; median over structures)0.595
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)F1 (attention; mean over structures)0.512±0.262
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)F1 (attention; median over structures)0.6
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)MCC (attention; mean over structures)0.520±0.263
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (1.6B) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)MCC (attention; median over structures)0.618
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)MCC (PLL-D; mean over assays)0.122±0.107
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs)0.854±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (FFT embedding; mean over random samplings of sequence pairs)0.944±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (FFT embedding; mean over random samplings of sequence pairs)0.873±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (FFT embedding; mean over random samplings of sequence pairs)0.745±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs)0.728±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (FFT embedding; mean over random samplings of sequence pairs)0.840±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (FFT embedding; mean over random samplings of sequence pairs)0.782±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (FFT embedding; mean over random samplings of sequence pairs)0.555±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs)0.889±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (CLS embedding; mean over random samplings of sequence pairs)0.964±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (CLS embedding; mean over random samplings of sequence pairs)0.901±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (CLS embedding; mean over random samplings of sequence pairs)0.802±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs)0.767±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (CLS embedding; mean over random samplings of sequence pairs)0.897±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (CLS embedding; mean over random samplings of sequence pairs)0.810±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningAIDO.RNA (650M) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (CLS embedding; mean over random samplings of sequence pairs)0.615±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (mean over assays)AUROC (WT-LLR; mean over assays)0.581±0.071
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (median over assays)AUROC (WT-LLR; median over assays)0.568
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (mean over assays)MCC (WT-LLR; mean over assays)0.125±0.105
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (median over assays)MCC (WT-LLR; median over assays)0.099
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (mean over assays)Absolute spearman correlation (WT-LLR; mean over assays)0.162±0.148
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (median over assays)Absolute spearman correlation (WT-LLR; median over assays)0.133
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)AUROC (PLL-D; mean over assays)0.551±0.045
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)AUROC (PLL-D; median over assays)0.541
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)MCC (PLL-D; mean over assays)0.076±0.069
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)MCC (PLL-D; median over assays)0.051
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)Absolute spearman correlation (PLL-D; mean over assays)0.100±0.092
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)Absolute spearman correlation (PLL-D; median over assays)0.081
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)F1 (attention; mean over structures)0.288±0.223
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)F1 (attention; median over structures)0.238
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)MCC (attention; mean over structures)0.290±0.221
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)MCC (attention; median over structures)0.236
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)F1 (attention; mean over structures)0.240±0.152
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)F1 (attention; median over structures)0.231
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)MCC (attention; mean over structures)0.243±0.152
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)MCC (attention; median over structures)0.237
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs)0.709±0.000
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (FFT embedding; mean over random samplings of sequence pairs)0.854±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (FFT embedding; mean over random samplings of sequence pairs)0.775±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (FFT embedding; mean over random samplings of sequence pairs)0.532±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs)0.572±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (FFT embedding; mean over random samplings of sequence pairs)0.765±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (FFT embedding; mean over random samplings of sequence pairs)0.713±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (FFT embedding; mean over random samplings of sequence pairs)0.367±0.011
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs)0.751±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (CLS embedding; mean over random samplings of sequence pairs)0.880±0.000
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (CLS embedding; mean over random samplings of sequence pairs)0.801±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (CLS embedding; mean over random samplings of sequence pairs)0.594±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs)0.612±0.000
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (CLS embedding; mean over random samplings of sequence pairs)0.784±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (CLS embedding; mean over random samplings of sequence pairs)0.728±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (CLS embedding; mean over random samplings of sequence pairs)0.404±0.002

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-8c69346a6089882b · All contested results · How findings are made and followed up