rewirebio.iobenchmarks
Contested result

DNABERT and UTR-LM: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks DNABERT above UTR-LM in 8 of 8 comparisons; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks UTR-LM above DNABERT in 25 of 26 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank DNABERT above UTR-LM in 9 of their 42 comparisons and UTR-LM higher in 25; 8 are ties. Of those papers, Zero-shot benchmarking of RNA language… ranks UTR-LM higher on its own (25 of 30).

Papers grouped by which system they rank higher: DNABERT or UTR-LMRank DNABERT higher: Benchmarking pre-trained genomic…. Rank UTR-LM higher: Zero-shot benchmarking of RNA language…. Mixed: none.Rank DNABERT higherMixedRank UTR-LM higherBenchmarkingpre-trained genomic…Benchmarking pre-trained genomic language models for RNA sequence-related predictive applicationsZero-shot benchmarkingof RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning
  • Rank DNABERT higher (1): Benchmarking pre-trained genomic…
  • Rank UTR-LM higher (1): Zero-shot benchmarking of RNA language…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores DNABERT or UTR-LM higherZero-shot benchmarking of RNA language…: DNABERT higher in 1, UTR-LM higher in 25, 4 ties. Benchmarking pre-trained genomic…: DNABERT higher in 8, UTR-LM higher in 0, 4 ties.Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning1 DNABERT higher4 tie425 UTR-LM higher25Benchmarking pre-trained genomic…Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications8 DNABERT higher84 tie4
DNABERT higherTieUTR-LM higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks DNABERT higherRanks UTR-LM higherTies
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOIAgrees1254
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and 1 more record · DOIAgrees804
Comparisons in Zero-shot benchmarking of RNA language… (30)
ProtocolDatasetMetricDNABERTUTR-LMHigher
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; mean over assays), unitless0.5350.551UTR-LM
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; median over assays), unitless0.5310.533Tie
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; mean over assays), unitless0.0540.071UTR-LM
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; median over assays), unitless0.0480.049UTR-LM
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; mean over assays), unitless0.0730.103UTR-LM
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; median over assays), unitless0.0680.067DNABERT
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.0110.029UTR-LM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless00.022UTR-LM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless-0.0080.016UTR-LM
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless-0.0060.006UTR-LM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.0030.016UTR-LM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless00.011UTR-LM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless-0.010.006UTR-LM
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless-0.0050UTR-LM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.5850.631UTR-LM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.7790.801UTR-LM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.7210.729UTR-LM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless0.3730.405UTR-LM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.3520.354Tie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.6370.65UTR-LM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.6670.669Tie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless-0.0020.077UTR-LM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.5910.649UTR-LM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.7860.813UTR-LM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.7260.739UTR-LM
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless0.3880.445UTR-LM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.3290.34UTR-LM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.6260.637UTR-LM
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.6670.667Tie
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless00.042UTR-LM
Comparisons in Benchmarking pre-trained genomic… (12)
ProtocolDatasetMetricDNABERTUTR-LMHigher
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsAccuracy (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.969DNABERT
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsF1 (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.969DNABERT
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsPrecision (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.97DNABERT
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsRecall (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.969DNABERT
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesAccuracy (NT-splice-sites-all, fine-tuned), fraction0.9830.976Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesF1 (NT-splice-sites-all, fine-tuned), fraction0.9830.976Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesPrecision (NT-splice-sites-all, fine-tuned), fraction0.9830.976Tie
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesRecall (NT-splice-sites-all, fine-tuned), fraction0.9830.976Tie
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsAccuracy (NT-splice-sites-donors, fine-tuned), fraction0.9850.974DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsF1 (NT-splice-sites-donors, fine-tuned), fraction0.9850.974DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsPrecision (NT-splice-sites-donors, fine-tuned), fraction0.9850.974DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsRecall (NT-splice-sites-donors, fine-tuned), fraction0.9850.974DNABERT

What disagrees

studies
2
Compared
DNABERT, UTR-LM
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-39d9092d64d1f720 · All contested results · How findings are made and followed up