Contested result

RNA-FM and SpliceBERT: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning disagree

Low severity
Not yet reviewed
two-study-conflict
unknown

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks SpliceBERT above RNA-FM in 3 of 3 comparisons; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks RNA-FM above SpliceBERT in 31 of 36 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank RNA-FM above SpliceBERT in 31 of their 51 comparisons and SpliceBERT higher in 8; 12 are ties. Of those papers, Benchmarking pre-trained genomic… ranks SpliceBERT higher on its own (3 of 15).

Rank RNA-FM higher (1): Zero-shot benchmarking of RNA language…
Rank SpliceBERT higher (1): Benchmarking pre-trained genomic…
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

RNA-FM higherTieSpliceBERT higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks RNA-FM higher	Ranks SpliceBERT higher	Ties
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOI	Agrees	31	5	0
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and 1 more record · DOI	Agrees	0	3	12

Comparisons in Zero-shot benchmarking of RNA language… (36)

Protocol	Dataset	Metric	RNA-FM	SpliceBERT	Higher
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	AUROC (WT-LLR; mean over assays), unitless	0.609	0.631	SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, AUC (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	AUROC (WT-LLR; median over assays), unitless	0.6	0.612	SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	MCC (WT-LLR; mean over assays), unitless	0.168	0.194	SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, MCC (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	MCC (WT-LLR; median over assays), unitless	0.172	0.176	SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	Absolute spearman correlation (WT-LLR; mean over assays), unitless	0.226	0.268	SpliceBERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), WT-LLR, SR (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	Absolute spearman correlation (WT-LLR; median over assays), unitless	0.207	0.202	RNA-FM
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	AUROC (PLL-D; mean over assays), unitless	0.569	0.55	RNA-FM
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	AUROC (PLL-D; median over assays), unitless	0.55	0.543	RNA-FM
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	MCC (PLL-D; mean over assays), unitless	0.095	0.071	RNA-FM
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	MCC (PLL-D; median over assays), unitless	0.075	0.063	RNA-FM
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	Absolute spearman correlation (PLL-D; mean over assays), unitless	0.14	0.097	RNA-FM
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)	RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)	Absolute spearman correlation (PLL-D; median over assays), unitless	0.103	0.079	RNA-FM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	F1 (attention; mean over structures), unitless	0.267	0.043	RNA-FM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	F1 (attention; median over structures), unitless	0.18	0.034	RNA-FM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	MCC (attention; mean over structures), unitless	0.273	0.033	RNA-FM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	MCC (attention; median over structures), unitless	0.186	0.022	RNA-FM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	F1 (attention; mean over structures), unitless	0.224	0.03	RNA-FM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	F1 (attention; median over structures), unitless	0.148	0.034	RNA-FM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	MCC (attention; mean over structures), unitless	0.231	0.02	RNA-FM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	MCC (attention; median over structures), unitless	0.164	0.025	RNA-FM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)	RfamSample (561 Rfam families) (Wang et al. 2026)	One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless	0.878	0.567	RNA-FM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)	RfamSample (561 Rfam families) (Wang et al. 2026)	AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless	0.953	0.762	RNA-FM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)	RfamSample (561 Rfam families) (Wang et al. 2026)	F1 (FFT embedding; mean over random samplings of sequence pairs), unitless	0.889	0.706	RNA-FM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)	RfamSample (561 Rfam families) (Wang et al. 2026)	MCC (FFT embedding; mean over random samplings of sequence pairs), unitless	0.771	0.341	RNA-FM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)	ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)	One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless	0.75	0.487	RNA-FM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)	ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)	AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless	0.872	0.731	RNA-FM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)	ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)	F1 (FFT embedding; mean over random samplings of sequence pairs), unitless	0.789	0.712	RNA-FM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)	ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)	MCC (FFT embedding; mean over random samplings of sequence pairs), unitless	0.568	0.312	RNA-FM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)	RfamSample (561 Rfam families) (Wang et al. 2026)	One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless	0.891	0.633	RNA-FM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)	RfamSample (561 Rfam families) (Wang et al. 2026)	AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless	0.93	0.804	RNA-FM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)	RfamSample (561 Rfam families) (Wang et al. 2026)	F1 (CLS embedding; mean over random samplings of sequence pairs), unitless	0.898	0.735	RNA-FM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)	RfamSample (561 Rfam families) (Wang et al. 2026)	MCC (CLS embedding; mean over random samplings of sequence pairs), unitless	0.803	0.426	RNA-FM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)	ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)	One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless	0.771	0.469	RNA-FM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)	ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)	AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless	0.861	0.727	RNA-FM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)	ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)	F1 (CLS embedding; mean over random samplings of sequence pairs), unitless	0.803	0.709	RNA-FM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)	ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)	MCC (CLS embedding; mean over random samplings of sequence pairs), unitless	0.635	0.308	RNA-FM

Comparisons in Benchmarking pre-trained genomic… (15)

Protocol	Dataset	Metric	RNA-FM	SpliceBERT	Higher
GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories), fraction	0.796	0.832	SpliceBERT
GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice-15class, averaged over categories), fraction	0.677	0.754	SpliceBERT
GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice-53class, averaged over categories), fraction	0.684	0.739	SpliceBERT
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Accuracy (NT-splice-sites-acceptors, fine-tuned), fraction	0.98	0.987	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	F1 (NT-splice-sites-acceptors, fine-tuned), fraction	0.98	0.987	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Precision (NT-splice-sites-acceptors, fine-tuned), fraction	0.98	0.987	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Recall (NT-splice-sites-acceptors, fine-tuned), fraction	0.98	0.987	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Accuracy (NT-splice-sites-all, fine-tuned), fraction	0.981	0.986	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	F1 (NT-splice-sites-all, fine-tuned), fraction	0.981	0.986	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Precision (NT-splice-sites-all, fine-tuned), fraction	0.981	0.986	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Recall (NT-splice-sites-all, fine-tuned), fraction	0.981	0.986	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Accuracy (NT-splice-sites-donors, fine-tuned), fraction	0.984	0.992	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	F1 (NT-splice-sites-donors, fine-tuned), fraction	0.984	0.992	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Precision (NT-splice-sites-donors, fine-tuned), fraction	0.984	0.992	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Recall (NT-splice-sites-donors, fine-tuned), fraction	0.984	0.992	Tie

What disagrees

studies: 2

Compared: RNA-FM, SpliceBERT
Study that disagrees
Other studies
Rule: two-study-conflict; unknown
Model identity: Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-575366601b1f3089 · All contested results · How findings are made and followed up