Contested result

RNA-MSM and UTR-LM: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks UTR-LM above RNA-MSM in 3 of 3 comparisons; other studies rank RNA-MSM above UTR-LM in 8 of 8.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Zero-shot benchmarking of RNA language… ranks RNA-MSM above UTR-LM in all 8 of its comparisons. Benchmarking pre-trained genomic… ranks UTR-LM above RNA-MSM in all 3 comparisons that are not ties (12 are ties).

Rank RNA-MSM higher (1): Zero-shot benchmarking of RNA language…
Rank UTR-LM higher (1): Benchmarking pre-trained genomic… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

RNA-MSM higherTieUTR-LM higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks RNA-MSM higher	Ranks UTR-LM higher	Ties
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and 1 more record · DOI	Disagrees	0	3	12
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOI	Agrees	8	0	0

Comparisons in Benchmarking pre-trained genomic… (15)

Protocol	Dataset	Metric	RNA-MSM	UTR-LM	Higher
GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories), fraction	0.809	0.833	UTR-LM
GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice-15class, averaged over categories), fraction	0.691	0.737	UTR-LM
GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)	GTEx-derived tissue-specific splice site usage dataset (You et al. 2026)	AUPRC (GTEx splicing task, Splice-53class, averaged over categories), fraction	0.696	0.724	UTR-LM
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Accuracy (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.969	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	F1 (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.969	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Precision (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.97	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Recall (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.969	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Accuracy (NT-splice-sites-all, fine-tuned), fraction	0.979	0.976	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	F1 (NT-splice-sites-all, fine-tuned), fraction	0.979	0.976	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Precision (NT-splice-sites-all, fine-tuned), fraction	0.979	0.976	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Recall (NT-splice-sites-all, fine-tuned), fraction	0.979	0.976	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Accuracy (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.974	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	F1 (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.974	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Precision (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.974	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Recall (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.974	Tie

Comparisons in Zero-shot benchmarking of RNA language… (8)

Protocol	Dataset	Metric	RNA-MSM	UTR-LM	Higher
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	F1 (attention; mean over structures), unitless	0.582	0.029	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	F1 (attention; median over structures), unitless	0.631	0.022	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	MCC (attention; mean over structures), unitless	0.588	0.016	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	MCC (attention; median over structures), unitless	0.634	0.006	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	F1 (attention; mean over structures), unitless	0.645	0.016	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	F1 (attention; median over structures), unitless	0.679	0.011	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	MCC (attention; mean over structures), unitless	0.654	0.006	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	MCC (attention; median over structures), unitless	0.69	0	RNA-MSM

What disagrees

Other studies: rank RNA-MSM above UTR-LM in 8 of 8 comparisons
This study: ranks UTR-LM above RNA-MSM in 3 of 3 comparisons

Compared: RNA-MSM, UTR-LM
Study that disagrees: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications, You et al. 2026, Supplementary Information
Other studies: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning, Zero-shot benchmarking of RNA language models: Supplementary Information
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (6)
Study	Tool	Protocol or benchmark	Metric	Value
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories)	0.8094(0.0010)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	UTRLM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice acceptors and donors (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice acceptors and donors, averaged over categories)	0.8334(0.0000)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-15class, averaged over categories)	0.6907(0.0044)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	UTRLM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-15class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-15class, averaged over categories)	0.7367(0.0000)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-53class, averaged over categories)	0.6955(0.0094)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	UTRLM fine-tuned with an appended convolutional layer (You et al. 2026)	GTEx tissue-specific splicing prediction, Splice-53class (You et al. 2026 Supplementary Table 5)	AUPRC (GTEx splicing task, Splice-53class, averaged over categories)	0.7244(0.0000)

In the other studies (16)
Study	Tool	Protocol or benchmark	Metric	Value
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.582±0.217
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.631
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.588±0.217
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.634
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.645±0.136
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.679
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.654±0.138
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.69
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	UTR_LM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.029±0.032
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	UTR_LM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.022
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	UTR_LM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.016±0.037
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	UTR_LM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.006
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	UTR_LM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.016±0.018
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	UTR_LM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.011
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	UTR_LM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.006±0.018
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	UTR_LM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-b1a83c26934d0af1 · All contested results · How findings are made and followed up