rewirebio.iobenchmarks
Contested result

RNAErnie and RiNALMo: RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Different tasks across studies

RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above RiNALMo in 3 of 4 comparisons; other studies rank RiNALMo above RNAErnie in 29 of 35.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Zero-shot benchmarking of RNA language… ranks RiNALMo above RNAErnie in 29 of its 36 comparisons and RNAErnie higher in 6; 1 is a tie. RNAGym fitness leaderboard ranks RNAErnie above RiNALMo in 3 of its 4 comparisons and RiNALMo higher in 1.

Papers grouped by which system they rank higher: RiNALMo or RNAErnieRank RiNALMo higher: Zero-shot benchmarking of RNA language…. Rank RNAErnie higher: RNAGym fitness leaderboard (disagrees). Mixed: none.Rank RiNALMo higherMixedRank RNAErnie higherZero-shot benchmarkingof RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md
  • Rank RiNALMo higher (1): Zero-shot benchmarking of RNA language…
  • Rank RNAErnie higher (1): RNAGym fitness leaderboard (disagrees)
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores RiNALMo or RNAErnie higherRNAGym fitness leaderboard (disagrees): RiNALMo higher in 1, RNAErnie higher in 3, 0 ties. Zero-shot benchmarking of RNA language…: RiNALMo higher in 29, RNAErnie higher in 6, 1 tie.Disagrees: RNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md1 RiNALMo higher13 RNAErnie higher3Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning29 RiNALMo higher291 tie6 RNAErnie higher6
RiNALMo higherTieRNAErnie higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks RiNALMo higherRanks RNAErnie higherTies
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md · RepositoryDisagrees130
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOIAgrees2961
Comparisons in RNAGym fitness leaderboard (4)
ProtocolDatasetMetricRiNALMoRNAErnieHigher
RNAGym v0.1.1 fitness, signed Spearman, AptamerRNAGym Aptamer DMS assays (2)Spearman (mean signed Spearman over the 2 Aptamer assays), unitless0.03560.0306RiNALMo
RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesRNAGym ncRNA DMS assays (31)Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless0.1390.194RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, RibozymeRNAGym Ribozyme DMS assays (26)Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless-0.00920.134RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, tRNARNAGym tRNA DMS assays (3)Spearman (mean signed Spearman over the 3 tRNA assays), unitless0.3910.416RNAErnie
Comparisons in Zero-shot benchmarking of RNA language… (36)
ProtocolDatasetMetricRiNALMoRNAErnieHigher
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (WT-LLR; mean over assays), unitless0.5880.581RiNALMo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (WT-LLR; median over assays), unitless0.5560.568RNAErnie
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (WT-LLR; mean over assays), unitless0.1410.125RiNALMo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (WT-LLR; median over assays), unitless0.1040.099RiNALMo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (WT-LLR; mean over assays), unitless0.1780.162RiNALMo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (WT-LLR; median over assays), unitless0.1140.133RNAErnie
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; mean over assays), unitless0.5840.551RiNALMo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; median over assays), unitless0.5710.541RiNALMo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; mean over assays), unitless0.1250.076RiNALMo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; median over assays), unitless0.1010.051RiNALMo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; mean over assays), unitless0.1680.1RiNALMo
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; median over assays), unitless0.1350.081RiNALMo
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.4140.288RiNALMo
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless0.4040.238RiNALMo
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless0.4230.29RiNALMo
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless0.4160.236RiNALMo
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.3650.24RiNALMo
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless0.3370.231RiNALMo
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless0.3750.243RiNALMo
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless0.3470.237RiNALMo
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.790.709RiNALMo
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.8960.854RiNALMo
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.8240.775RiNALMo
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless0.6570.532RiNALMo
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.6210.572RiNALMo
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.7050.765RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.6720.713RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless0.4350.367RiNALMo
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.8030.751RiNALMo
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.9010.88RiNALMo
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.8360.801RiNALMo
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless0.6670.594RiNALMo
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.6590.612RiNALMo
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.7210.784RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.6840.728RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless0.4010.404Tie

What disagrees

Other studies
rank RiNALMo above RNAErnie in 29 of 35 comparisons
This study
ranks RNAErnie above RiNALMo in 3 of 4 comparisons
Compared
RNAErnie, RiNALMo
Study that disagrees
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md
Other studies
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning, Zero-shot benchmarking of RNA language models: Supplementary Information
Rule
Ordering reversal; different tasks across studies
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (8)
StudyToolProtocol or benchmarkMetricValue
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRiNALMo (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, AptamerSpearman (mean signed Spearman over the 2 Aptamer assays)0.0356
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRiNALMo (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesSpearman (macro-mean over 3 ncRNA categories (equal category weights))0.139
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRiNALMo (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, RibozymeSpearman (mean signed Spearman over the 26 Ribozyme assays)-0.0092
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRiNALMo (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, tRNASpearman (mean signed Spearman over the 3 tRNA assays)0.391
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRNA-ERNIE (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, AptamerSpearman (mean signed Spearman over the 2 Aptamer assays)0.0306
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRNA-ERNIE (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesSpearman (macro-mean over 3 ncRNA categories (equal category weights))0.194
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRNA-ERNIE (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, RibozymeSpearman (mean signed Spearman over the 26 Ribozyme assays)0.134
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdRNA-ERNIE (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, tRNASpearman (mean signed Spearman over the 3 tRNA assays)0.416
In the other studies (72 results)
In the other studies
StudyToolProtocol or benchmarkMetricValue
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)F1 (attention; mean over structures)0.414±0.255
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)F1 (attention; median over structures)0.404
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)MCC (attention; mean over structures)0.423±0.257
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)MCC (attention; median over structures)0.416
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)F1 (attention; mean over structures)0.365±0.214
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)F1 (attention; median over structures)0.337
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)MCC (attention; mean over structures)0.375±0.222
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)MCC (attention; median over structures)0.347
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs)0.790±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (FFT embedding; mean over random samplings of sequence pairs)0.896±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (FFT embedding; mean over random samplings of sequence pairs)0.824±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (FFT embedding; mean over random samplings of sequence pairs)0.657±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (FFT embedding; mean over random samplings of sequence pairs)0.705±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs)0.803±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (CLS embedding; mean over random samplings of sequence pairs)0.901±0.000
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (CLS embedding; mean over random samplings of sequence pairs)0.836±0.000
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (CLS embedding; mean over random samplings of sequence pairs)0.667±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (giga) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (CLS embedding; mean over random samplings of sequence pairs)0.721±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (mean over assays)AUROC (WT-LLR; mean over assays)0.588±0.073
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (median over assays)AUROC (WT-LLR; median over assays)0.556
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (mean over assays)MCC (WT-LLR; mean over assays)0.141±0.108
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (median over assays)MCC (WT-LLR; median over assays)0.104
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (mean over assays)Absolute spearman correlation (WT-LLR; mean over assays)0.178±0.152
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (median over assays)Absolute spearman correlation (WT-LLR; median over assays)0.114
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)AUROC (PLL-D; mean over assays)0.584±0.073
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)AUROC (PLL-D; median over assays)0.571
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)MCC (PLL-D; mean over assays)0.125±0.108
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)MCC (PLL-D; median over assays)0.101
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)Absolute spearman correlation (PLL-D; mean over assays)0.168±0.149
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)Absolute spearman correlation (PLL-D; median over assays)0.135
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (mega) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (FFT embedding; mean over random samplings of sequence pairs)0.672±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (micro) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs)0.621±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (micro) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (FFT embedding; mean over random samplings of sequence pairs)0.705±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (micro) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (FFT embedding; mean over random samplings of sequence pairs)0.672±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (micro) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (FFT embedding; mean over random samplings of sequence pairs)0.435±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (micro) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs)0.659±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRiNALMo (micro) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (CLS embedding; mean over random samplings of sequence pairs)0.684±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (mean over assays)AUROC (WT-LLR; mean over assays)0.581±0.071
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (median over assays)AUROC (WT-LLR; median over assays)0.568
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (mean over assays)MCC (WT-LLR; mean over assays)0.125±0.105
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (median over assays)MCC (WT-LLR; median over assays)0.099
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (mean over assays)Absolute spearman correlation (WT-LLR; mean over assays)0.162±0.148
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (median over assays)Absolute spearman correlation (WT-LLR; median over assays)0.133
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)AUROC (PLL-D; mean over assays)0.551±0.045
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)AUROC (PLL-D; median over assays)0.541
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)MCC (PLL-D; mean over assays)0.076±0.069
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)MCC (PLL-D; median over assays)0.051
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)Absolute spearman correlation (PLL-D; mean over assays)0.100±0.092
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)Absolute spearman correlation (PLL-D; median over assays)0.081
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)F1 (attention; mean over structures)0.288±0.223
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)F1 (attention; median over structures)0.238
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)MCC (attention; mean over structures)0.290±0.221
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)MCC (attention; median over structures)0.236
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)F1 (attention; mean over structures)0.240±0.152
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)F1 (attention; median over structures)0.231
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)MCC (attention; mean over structures)0.243±0.152
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)MCC (attention; median over structures)0.237
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs)0.709±0.000
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (FFT embedding; mean over random samplings of sequence pairs)0.854±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (FFT embedding; mean over random samplings of sequence pairs)0.775±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (FFT embedding; mean over random samplings of sequence pairs)0.532±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs)0.572±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (FFT embedding; mean over random samplings of sequence pairs)0.765±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (FFT embedding; mean over random samplings of sequence pairs)0.713±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (FFT embedding; mean over random samplings of sequence pairs)0.367±0.011
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs)0.751±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (CLS embedding; mean over random samplings of sequence pairs)0.880±0.000
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (CLS embedding; mean over random samplings of sequence pairs)0.801±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)MCC (CLS embedding; mean over random samplings of sequence pairs)0.594±0.002
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs)0.612±0.000
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)AUROC (CLS embedding; mean over random samplings of sequence pairs)0.784±0.001
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAErnie (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)F1 (CLS embedding; mean over random samplings of sequence pairs)0.728±0.001

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-0cbdc15cd5769e28 · All contested results · How findings are made and followed up