rewirebio.iobenchmarks
Contested result

RNAErnie and SpliceBERT: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks SpliceBERT above RNAErnie in 3 of 3 comparisons; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks RNAErnie above SpliceBERT in 26 of 33 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank RNAErnie above SpliceBERT in 26 of their 51 comparisons and SpliceBERT higher in 10; 15 are ties. Of those papers, Benchmarking pre-trained genomic… ranks SpliceBERT higher on its own (3 of 15).

Papers grouped by which system they rank higher: RNAErnie or SpliceBERTRank RNAErnie higher: Zero-shot benchmarking of RNA language…. Rank SpliceBERT higher: Benchmarking pre-trained genomic…. Mixed: none.Rank RNAErnie higherMixedRank SpliceBERT higherZero-shot benchmarkingof RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningBenchmarkingpre-trained genomic…Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications
  • Rank RNAErnie higher (1): Zero-shot benchmarking of RNA language…
  • Rank SpliceBERT higher (1): Benchmarking pre-trained genomic…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores RNAErnie or SpliceBERT higherZero-shot benchmarking of RNA language…: RNAErnie higher in 26, SpliceBERT higher in 7, 3 ties. Benchmarking pre-trained genomic…: RNAErnie higher in 0, SpliceBERT higher in 3, 12 ties.Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning26 RNAErnie higher263 tie7 SpliceBERT higher7Benchmarking pre-trained genomic…Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications12 tie123 SpliceBERT higher3
RNAErnie higherTieSpliceBERT higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks RNAErnie higherRanks SpliceBERT higherTies
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOIAgrees2673
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and 1 more record · DOIAgrees0312
Comparisons in Zero-shot benchmarking of RNA language… (36)
ProtocolDatasetMetricRNAErnieSpliceBERTHigher
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (WT-LLR; mean over assays), unitless0.5810.631SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (WT-LLR; median over assays), unitless0.5680.612SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (WT-LLR; mean over assays), unitless0.1250.194SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (WT-LLR; median over assays), unitless0.0990.176SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (WT-LLR; mean over assays), unitless0.1620.268SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (WT-LLR; median over assays), unitless0.1330.202SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; mean over assays), unitless0.5510.55Tie
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; median over assays), unitless0.5410.543Tie
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; mean over assays), unitless0.0760.071RNAErnie
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; median over assays), unitless0.0510.063SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; mean over assays), unitless0.10.097RNAErnie
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; median over assays), unitless0.0810.079RNAErnie
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.2880.043RNAErnie
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless0.2380.034RNAErnie
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless0.290.033RNAErnie
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless0.2360.022RNAErnie
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.240.03RNAErnie
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless0.2310.034RNAErnie
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless0.2430.02RNAErnie
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless0.2370.025RNAErnie
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.7090.567RNAErnie
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.8540.762RNAErnie
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.7750.706RNAErnie
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless0.5320.341RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.5720.487RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.7650.731RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.7130.712Tie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless0.3670.312RNAErnie
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.7510.633RNAErnie
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.880.804RNAErnie
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.8010.735RNAErnie
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless0.5940.426RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.6120.469RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.7840.727RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.7280.709RNAErnie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless0.4040.308RNAErnie
Comparisons in Benchmarking pre-trained genomic… (15)
ProtocolDatasetMetricRNAErnieSpliceBERTHigher
GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories), fraction0.8060.832SpliceBERT
GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)AUPRC (GTEx splicing task, Splice-15class, averaged over categories), fraction0.6870.754SpliceBERT
GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)AUPRC (GTEx splicing task, Splice-53class, averaged over categories), fraction0.6850.739SpliceBERT
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsAccuracy (NT-splice-sites-acceptors, fine-tuned), fraction0.9790.987Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsF1 (NT-splice-sites-acceptors, fine-tuned), fraction0.9790.987Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsPrecision (NT-splice-sites-acceptors, fine-tuned), fraction0.9790.987Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsRecall (NT-splice-sites-acceptors, fine-tuned), fraction0.9790.987Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesAccuracy (NT-splice-sites-all, fine-tuned), fraction0.9810.986Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesF1 (NT-splice-sites-all, fine-tuned), fraction0.9810.986Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesPrecision (NT-splice-sites-all, fine-tuned), fraction0.9810.986Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesRecall (NT-splice-sites-all, fine-tuned), fraction0.9810.986Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsAccuracy (NT-splice-sites-donors, fine-tuned), fraction0.9830.992Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsF1 (NT-splice-sites-donors, fine-tuned), fraction0.9830.992Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsPrecision (NT-splice-sites-donors, fine-tuned), fraction0.9830.992Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsRecall (NT-splice-sites-donors, fine-tuned), fraction0.9830.992Tie

What disagrees

studies
2
Compared
RNAErnie, SpliceBERT
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-89c4be1cc4090cf4 · All contested results · How findings are made and followed up