Contested result

RNA-MSM and SpliceBERT: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks SpliceBERT above RNA-MSM in 7 of 7 comparisons; other studies rank RNA-MSM above SpliceBERT in 8 of 8.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Zero-shot benchmarking of RNA language… ranks RNA-MSM above SpliceBERT in all 8 of its comparisons. Benchmarking pre-trained genomic… ranks SpliceBERT above RNA-MSM in all 7 comparisons that are not ties (8 are ties).

Rank RNA-MSM higher (1): Zero-shot benchmarking of RNA language…
Rank SpliceBERT higher (1): Benchmarking pre-trained genomic… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

RNA-MSM higherTieSpliceBERT higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks RNA-MSM higher	Ranks SpliceBERT higher	Ties
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and 1 more record · DOI	Disagrees	0	7	8
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOI	Agrees	8	0	0

Comparisons in Benchmarking pre-trained genomic… (15)

Protocol	Dataset	Metric	RNA-MSM	SpliceBERT	Higher
GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories), fraction	0.809	0.832	SpliceBERT
GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice-15class, averaged over categories), fraction	0.691	0.754	SpliceBERT
GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice-53class, averaged over categories), fraction	0.696	0.739	SpliceBERT
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Accuracy (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.987	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	F1 (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.987	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Precision (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.987	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Recall (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.987	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Accuracy (NT-splice-sites-all, fine-tuned), fraction	0.979	0.986	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	F1 (NT-splice-sites-all, fine-tuned), fraction	0.979	0.986	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Precision (NT-splice-sites-all, fine-tuned), fraction	0.979	0.986	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Recall (NT-splice-sites-all, fine-tuned), fraction	0.979	0.986	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Accuracy (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.992	SpliceBERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	F1 (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.992	SpliceBERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Precision (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.992	SpliceBERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Recall (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.992	SpliceBERT

Comparisons in Zero-shot benchmarking of RNA language… (8)

Protocol	Dataset	Metric	RNA-MSM	SpliceBERT	Higher
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	F1 (attention; mean over structures), unitless	0.582	0.043	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	F1 (attention; median over structures), unitless	0.631	0.034	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	MCC (attention; mean over structures), unitless	0.588	0.033	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	MCC (attention; median over structures), unitless	0.634	0.022	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	F1 (attention; mean over structures), unitless	0.645	0.03	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	F1 (attention; median over structures), unitless	0.679	0.034	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	MCC (attention; mean over structures), unitless	0.654	0.02	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	MCC (attention; median over structures), unitless	0.69	0.025	RNA-MSM

What disagrees

Other studies: rank RNA-MSM above SpliceBERT in 8 of 8 comparisons
This study: ranks SpliceBERT above RNA-MSM in 7 of 7 comparisons

Compared: RNA-MSM, SpliceBERT
Study that disagrees: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications, You et al. 2026, Supplementary Information
Other studies: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning, Zero-shot benchmarking of RNA language models: Supplementary Information
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (14)
Study	Tool	Protocol or benchmark	Metric	Value
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Accuracy (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	F1 (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Precision (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Recall (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Accuracy (NT-splice-sites-donors, fine-tuned)	0.9915(0.0010)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	F1 (NT-splice-sites-donors, fine-tuned)	0.9915(0.0010)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Precision (NT-splice-sites-donors, fine-tuned)	0.9915(0.0009)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Recall (NT-splice-sites-donors, fine-tuned)	0.9915(0.0010)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories)	0.8094(0.0010)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories)	0.8317(0.0096)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-15class, averaged over categories)	0.6907(0.0044)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-15class, averaged over categories)	0.7538(0.0014)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-53class, averaged over categories)	0.6955(0.0094)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-53class, averaged over categories)	0.7387(0.0020)

In the other studies (16)
Study	Tool	Protocol or benchmark	Metric	Value
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.582±0.217
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.631
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.588±0.217
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.634
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.645±0.136
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.679
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.654±0.138
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.69
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.043±0.039
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.034
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.033±0.045
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.022
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.030±0.023
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.034
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.020±0.026
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.025

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-47eecbc09b367751 · All contested results · How findings are made and followed up