rewirebio.iobenchmarks
Contested result

DNABERT and RNA-MSM: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks DNABERT above RNA-MSM in 4 of 4 comparisons; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks RNA-MSM above DNABERT in 8 of 8 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank DNABERT above RNA-MSM in 4 of their 20 comparisons and RNA-MSM higher in 8; 8 are ties. Of those papers, Zero-shot benchmarking of RNA language… ranks RNA-MSM higher on its own (8 of 8).

Papers grouped by which system they rank higher: DNABERT or RNA-MSMRank DNABERT higher: Benchmarking pre-trained genomic…. Rank RNA-MSM higher: Zero-shot benchmarking of RNA language…. Mixed: none.Rank DNABERT higherMixedRank RNA-MSM higherBenchmarkingpre-trained genomic…Benchmarking pre-trained genomic language models for RNA sequence-related predictive applicationsZero-shot benchmarkingof RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning
  • Rank DNABERT higher (1): Benchmarking pre-trained genomic…
  • Rank RNA-MSM higher (1): Zero-shot benchmarking of RNA language…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores DNABERT or RNA-MSM higherBenchmarking pre-trained genomic…: DNABERT higher in 4, RNA-MSM higher in 0, 8 ties. Zero-shot benchmarking of RNA language…: DNABERT higher in 0, RNA-MSM higher in 8, 0 ties.Benchmarking pre-trained genomic…Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications4 DNABERT higher48 tie8Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning8 RNA-MSM higher8
DNABERT higherTieRNA-MSM higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks DNABERT higherRanks RNA-MSM higherTies
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and 1 more record · DOIAgrees408
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOIAgrees080
Comparisons in Benchmarking pre-trained genomic… (12)
ProtocolDatasetMetricDNABERTRNA-MSMHigher
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsAccuracy (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.978Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsF1 (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.978Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsPrecision (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.978Tie
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsRecall (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.978Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesAccuracy (NT-splice-sites-all, fine-tuned), fraction0.9830.979Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesF1 (NT-splice-sites-all, fine-tuned), fraction0.9830.979Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesPrecision (NT-splice-sites-all, fine-tuned), fraction0.9830.979Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesRecall (NT-splice-sites-all, fine-tuned), fraction0.9830.979Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsAccuracy (NT-splice-sites-donors, fine-tuned), fraction0.9850.974DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsF1 (NT-splice-sites-donors, fine-tuned), fraction0.9850.974DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsPrecision (NT-splice-sites-donors, fine-tuned), fraction0.9850.974DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsRecall (NT-splice-sites-donors, fine-tuned), fraction0.9850.974DNABERT
Comparisons in Zero-shot benchmarking of RNA language… (8)
ProtocolDatasetMetricDNABERTRNA-MSMHigher
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.0110.582RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless00.631RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless-0.0080.588RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless-0.0060.634RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.0030.645RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless00.679RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless-0.010.654RNA-MSM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless-0.0050.69RNA-MSM

What disagrees

studies
2
Compared
DNABERT, RNA-MSM
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-1be866f365446e7a · All contested results · How findings are made and followed up