Contested result

DNABERT and RNA-MSM: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks DNABERT above RNA-MSM in 4 of 4 comparisons; other studies rank RNA-MSM above DNABERT in 8 of 8.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Zero-shot benchmarking of RNA language… ranks RNA-MSM above DNABERT in all 8 of its comparisons. Benchmarking pre-trained genomic… ranks DNABERT above RNA-MSM in all 4 comparisons that are not ties (8 are ties).

Rank RNA-MSM higher (1): Zero-shot benchmarking of RNA language…
Rank DNABERT higher (1): Benchmarking pre-trained genomic… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

RNA-MSM higherTieDNABERT higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks RNA-MSM higher	Ranks DNABERT higher	Ties
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and 1 more record · DOI	Disagrees	0	4	8
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOI	Agrees	8	0	0

Comparisons in Benchmarking pre-trained genomic… (12)

Protocol	Dataset	Metric	RNA-MSM	DNABERT	Higher
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Accuracy (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.983	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	F1 (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.983	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Precision (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.983	Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, acceptors	Recall (NT-splice-sites-acceptors, fine-tuned), fraction	0.978	0.983	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Accuracy (NT-splice-sites-all, fine-tuned), fraction	0.979	0.983	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	F1 (NT-splice-sites-all, fine-tuned), fraction	0.979	0.983	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Precision (NT-splice-sites-all, fine-tuned), fraction	0.979	0.983	Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, all sites	Recall (NT-splice-sites-all, fine-tuned), fraction	0.979	0.983	Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Accuracy (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.985	DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	F1 (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.985	DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Precision (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.985	DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Nucleotide Transformer benchmark splice-site dataset, donors	Recall (NT-splice-sites-donors, fine-tuned), fraction	0.974	0.985	DNABERT

Comparisons in Zero-shot benchmarking of RNA language… (8)

Protocol	Dataset	Metric	RNA-MSM	DNABERT	Higher
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	F1 (attention; mean over structures), unitless	0.582	0.017	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	F1 (attention; median over structures), unitless	0.631	0.01	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	MCC (attention; mean over structures), unitless	0.588	0.001	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	TS test set (70 PDB structures) (Wang et al. 2026)	MCC (attention; median over structures), unitless	0.634	0.003	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	F1 (attention; mean over structures), unitless	0.645	0.011	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	F1 (attention; median over structures), unitless	0.679	0.011	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	MCC (attention; mean over structures), unitless	0.654	-0.002	RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	TS-Hard test set (15 PDB structures) (Wang et al. 2026)	MCC (attention; median over structures), unitless	0.69	0.003	RNA-MSM

What disagrees

Other studies: rank RNA-MSM above DNABERT in 8 of 8 comparisons
This study: ranks DNABERT above RNA-MSM in 4 of 4 comparisons

Compared: DNABERT, RNA-MSM
Study that disagrees: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications, You et al. 2026, Supplementary Information
Other studies: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning, Zero-shot benchmarking of RNA language models: Supplementary Information
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (8)
Study	Tool	Protocol or benchmark	Metric	Value
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	DNABERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Accuracy (NT-splice-sites-donors, fine-tuned)	0.9851(0.0007)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	DNABERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	F1 (NT-splice-sites-donors, fine-tuned)	0.9851(0.0007)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	DNABERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Precision (NT-splice-sites-donors, fine-tuned)	0.9851(0.0007)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	DNABERT fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Recall (NT-splice-sites-donors, fine-tuned)	0.9851(0.0007)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Accuracy (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	F1 (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Precision (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications; You et al. 2026, Supplementary Information	RNAMSM fine-tuned with an appended convolutional layer (You et al. 2026)	NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)	Recall (NT-splice-sites-donors, fine-tuned)	0.9735(0.0005)

In the other studies (16)
Study	Tool	Protocol or benchmark	Metric	Value
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	DNABERT (k=6) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.017±0.021
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	DNABERT (k=6) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.01
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	DNABERT (k=6) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.001±0.024
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	DNABERT (k=6) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.003
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	DNABERT (k=6) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.011±0.011
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	DNABERT (k=6) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.011
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	DNABERT (k=6) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	-0.002±0.018
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	DNABERT (k=6) (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.003
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.582±0.217
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.631
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.588±0.217
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.634
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)	F1 (attention; mean over structures)	0.645±0.136
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)	F1 (attention; median over structures)	0.679
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)	MCC (attention; mean over structures)	0.654±0.138
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning	RNA-MSM (zero-shot, Wang et al. 2026)	Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)	MCC (attention; median over structures)	0.69

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-0ad90aa64c23cba9 · All contested results · How findings are made and followed up