rewirebio.iobenchmarks
Contested result

DNABERT and RNABERT: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks DNABERT above RNABERT in 12 of 12 comparisons; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks RNABERT above DNABERT in 24 of 26 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank DNABERT above RNABERT in 14 of their 42 comparisons and RNABERT higher in 24; 4 are ties. Of those papers, Zero-shot benchmarking of RNA language… ranks RNABERT higher on its own (24 of 30).

Papers grouped by which system they rank higher: DNABERT or RNABERTRank DNABERT higher: Benchmarking pre-trained genomic…. Rank RNABERT higher: Zero-shot benchmarking of RNA language…. Mixed: none.Rank DNABERT higherMixedRank RNABERT higherBenchmarkingpre-trained genomic…Benchmarking pre-trained genomic language models for RNA sequence-related predictive applicationsZero-shot benchmarkingof RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning
  • Rank DNABERT higher (1): Benchmarking pre-trained genomic…
  • Rank RNABERT higher (1): Zero-shot benchmarking of RNA language…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores DNABERT or RNABERT higherZero-shot benchmarking of RNA language…: DNABERT higher in 2, RNABERT higher in 24, 4 ties. Benchmarking pre-trained genomic…: DNABERT higher in 12, RNABERT higher in 0, 0 ties.Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning2 DNABERT higher4 tie424 RNABERT higher24Benchmarking pre-trained genomic…Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications12 DNABERT higher12
DNABERT higherTieRNABERT higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks DNABERT higherRanks RNABERT higherTies
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOIAgrees2244
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and 1 more record · DOIAgrees1200
Comparisons in Zero-shot benchmarking of RNA language… (30)
ProtocolDatasetMetricDNABERTRNABERTHigher
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; mean over assays), unitless0.5350.544RNABERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)AUROC (PLL-D; median over assays), unitless0.5310.534Tie
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; mean over assays), unitless0.0540.065RNABERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)MCC (PLL-D; median over assays), unitless0.0480.047DNABERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; mean over assays), unitless0.0730.09RNABERT
Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)RNAGym ncRNA DMS assays (31 assays) (Wang et al. 2026)Absolute spearman correlation (PLL-D; median over assays), unitless0.0680.067DNABERT
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.0110.022RNABERT
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, F1 score (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless00Tie
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (mean over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless-0.0080.01RNABERT
Wang et al. 2026 zero-shot secondary structure, TS test set (70 PDB structures), attention, MCC (median over structures)TS test set (70 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless-0.006-0.001RNABERT
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; mean over structures), unitless0.0030.014RNABERT
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, F1 score (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)F1 (attention; median over structures), unitless00Tie
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (mean over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; mean over structures), unitless-0.010.007RNABERT
Wang et al. 2026 zero-shot secondary structure, TS-Hard test set (15 PDB structures), attention, MCC (median over structures)TS-Hard test set (15 PDB structures) (Wang et al. 2026)MCC (attention; median over structures), unitless-0.005-0.001RNABERT
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.5850.724RNABERT
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.7790.777Tie
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.7210.794RNABERT
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless0.3730.586RNABERT
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (FFT embedding; mean over random samplings of sequence pairs), unitless0.3520.585RNABERT
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (FFT embedding; mean over random samplings of sequence pairs), unitless0.6370.732RNABERT
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (FFT embedding; mean over random samplings of sequence pairs), unitless0.6670.719RNABERT
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), FFT embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (FFT embedding; mean over random samplings of sequence pairs), unitless-0.0020.363RNABERT
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.5910.73RNABERT
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.7860.831RNABERT
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.7260.797RNABERT
Wang et al. 2026 zero-shot family classification, RfamSample (561 Rfam families), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)RfamSample (561 Rfam families) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless0.3880.588RNABERT
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, 1 − OR (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)One minus overlap ratio (CLS embedding; mean over random samplings of sequence pairs), unitless0.3290.561RNABERT
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, AUC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)AUROC (CLS embedding; mean over random samplings of sequence pairs), unitless0.6260.741RNABERT
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, F1 (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)F1 (CLS embedding; mean over random samplings of sequence pairs), unitless0.6670.71RNABERT
Wang et al. 2026 zero-shot RNA type classification, ArchiveII-Nr (1273 RNAs), CLS embedding, MCC (mean over repeated random samplings of sequence pairs)ArchiveII-Nr (1273 RNAs) (Wang et al. 2026)MCC (CLS embedding; mean over random samplings of sequence pairs), unitless00.35RNABERT
Comparisons in Benchmarking pre-trained genomic… (12)
ProtocolDatasetMetricDNABERTRNABERTHigher
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsAccuracy (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.959DNABERT
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsF1 (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.959DNABERT
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsPrecision (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.959DNABERT
NT-splice-sites-acceptors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, acceptorsRecall (NT-splice-sites-acceptors, fine-tuned), fraction0.9830.959DNABERT
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesAccuracy (NT-splice-sites-all, fine-tuned), fraction0.9830.969DNABERT
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesF1 (NT-splice-sites-all, fine-tuned), fraction0.9830.969DNABERT
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesPrecision (NT-splice-sites-all, fine-tuned), fraction0.9830.969DNABERT
NT-splice-sites-all splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, all sitesRecall (NT-splice-sites-all, fine-tuned), fraction0.9830.969DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsAccuracy (NT-splice-sites-donors, fine-tuned), fraction0.9850.967DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsF1 (NT-splice-sites-donors, fine-tuned), fraction0.9850.967DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsPrecision (NT-splice-sites-donors, fine-tuned), fraction0.9850.967DNABERT
NT-splice-sites-donors splice-site classification, fine-tuned models (You et al. 2026 Supplementary Table 4)Nucleotide Transformer benchmark splice-site dataset, donorsRecall (NT-splice-sites-donors, fine-tuned), fraction0.9850.967DNABERT

What disagrees

studies
2
Compared
DNABERT, RNABERT
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-b7026f393dee36cc · All contested results · How findings are made and followed up