Contested result

RNA-MSM and SpliceBERT: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks RNA-MSM above SpliceBERT in 8 of 8 comparisons; other studies rank SpliceBERT above RNA-MSM in 7 of 7.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Benchmarking pre-trained genomic… ranks SpliceBERT above RNA-MSM in all 7 comparisons that are not ties (8 are ties). Zero-shot benchmarking of RNA language… ranks RNA-MSM above SpliceBERT in all 8 of its comparisons.

Rank SpliceBERT higher (1): Benchmarking pre-trained genomic…
Rank RNA-MSM higher (1): Zero-shot benchmarking of RNA language… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

SpliceBERT higherTieRNA-MSM higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks SpliceBERT higher	Ranks RNA-MSM higher	Ties
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOI	Disagrees	0	8	0
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and 1 more record · DOI	Agrees	7	0	8

Comparisons in Zero-shot benchmarking of RNA language… (8)

Protocol	Dataset	Metric	SpliceBERT	RNA-MSM	Higher
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	F1 (attention; mean over structures), unitless	0.043	0.582	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	F1 (attention; median over structures), unitless	0.034	0.631	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	MCC (attention; mean over structures), unitless	0.033	0.588	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	MCC (attention; median over structures), unitless	0.022	0.634	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	F1 (attention; mean over structures), unitless	0.03	0.645	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	F1 (attention; median over structures), unitless	0.034	0.679	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	MCC (attention; mean over structures), unitless	0.02	0.654	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	MCC (attention; median over structures), unitless	0.025	0.69	RNA-MSM

Comparisons in Benchmarking pre-trained genomic… (15)

Protocol	Dataset	Metric	SpliceBERT	RNA-MSM	Higher
GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories), fraction	0.832	0.809	SpliceBERT
GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice-15class, averaged over categories), fraction	0.754	0.691	SpliceBERT
GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice-53class, averaged over categories), fraction	0.739	0.696	SpliceBERT
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Accuracy (NT-splice-sites-acceptors, fine-tuned), fraction	0.987	0.978	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	F1 (NT-splice-sites-acceptors, fine-tuned), fraction	0.987	0.978	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Precision (NT-splice-sites-acceptors, fine-tuned), fraction	0.987	0.978	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Recall (NT-splice-sites-acceptors, fine-tuned), fraction	0.987	0.978	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Accuracy (NT-splice-sites-all, fine-tuned), fraction	0.986	0.979	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	F1 (NT-splice-sites-all, fine-tuned), fraction	0.986	0.979	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Precision (NT-splice-sites-all, fine-tuned), fraction	0.986	0.979	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Recall (NT-splice-sites-all, fine-tuned), fraction	0.986	0.979	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Accuracy (NT-splice-sites-donors, fine-tuned), fraction	0.992	0.974	SpliceBERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	F1 (NT-splice-sites-donors, fine-tuned), fraction	0.992	0.974	SpliceBERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Precision (NT-splice-sites-donors, fine-tuned), fraction	0.992	0.974	SpliceBERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Recall (NT-splice-sites-donors, fine-tuned), fraction	0.992	0.974	SpliceBERT

What disagrees

Other studies: rank SpliceBERT above RNA-MSM in 7 of 7 comparisons
This study: ranks RNA-MSM above SpliceBERT in 8 of 8 comparisons

Compared: RNA-MSM, SpliceBERT
Study that disagrees: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning, Zero-shot benchmarking of RNA language models: Supplementary Information
Other studies: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications, You et al. 2026, Supplementary Information
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (16)
Study	Tool	Protocol or benchmark	Metric	Value
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.582±0.217
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.631
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.588±0.217
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.634
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.645±0.136
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.679
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.654±0.138
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.69
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.043±0.039
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.034
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.033±0.045
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.022
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.030±0.023
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.034
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.020±0.026
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	SpliceBERT (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.025

In the other studies (14)
Study	Tool	Protocol or benchmark	Metric	Value
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Accuracy (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	F1 (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Precision (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Recall (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Accuracy (NT-splice-sites-donors, fine-tuned)	0.9915(0.0010)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	F1 (NT-splice-sites-donors, fine-tuned)	0.9915(0.0010)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Precision (NT-splice-sites-donors, fine-tuned)	0.9915(0.0009)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Recall (NT-splice-sites-donors, fine-tuned)	0.9915(0.0010)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories)	0.8094(0.0010)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories)	0.8317(0.0096)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-15class, averaged over categories)	0.6907(0.0044)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-15class, averaged over categories)	0.7538(0.0014)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-53class, averaged over categories)	0.6955(0.0094)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	SpliceBERT fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-53class, averaged over categories)	0.7387(0.0020)

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-d35b238ad04a888e · All contested results · How findings are made and followed up