rewirebio.iobenchmarks
Contested result

DNABERT-2 and Evo: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above DNABERT-2 in 6 of 6 comparisons; 2 of 2 other studies rank DNABERT-2 above Evo.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank DNABERT-2 above Evo in 45 of their 55 comparisons and Evo higher in 10. Zero-shot benchmarking of RNA language… ranks Evo above DNABERT-2 in all 6 of its comparisons.

Papers grouped by which system they rank higher: DNABERT-2 or EvoRank DNABERT-2 higher: GENEB leaderboard, BacBench LICENSE file at commit 64b7d45. Rank Evo higher: Zero-shot benchmarking of RNA language… (disagrees). Mixed: none.Rank DNABERT-2 higherMixedRank Evo higherGENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)BacBench LICENSE fileat commit 64b7d45BacBench LICENSE file at commit 64b7d45Zero-shot benchmarkingof RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning
  • Rank DNABERT-2 higher (2): GENEB leaderboard, BacBench LICENSE file at commit 64b7d45
  • Rank Evo higher (1): Zero-shot benchmarking of RNA language… (disagrees)
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores DNABERT-2 or Evo higherZero-shot benchmarking of RNA language… (disagrees): DNABERT-2 higher in 0, Evo higher in 6, 0 ties. GENEB leaderboard: DNABERT-2 higher in 35, Evo higher in 10, 0 ties. BacBench LICENSE file at commit 64b7d45: DNABERT-2 higher in 10, Evo higher in 0, 0 ties.Disagrees: Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning6 Evo higher6GENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)35 DNABERT-2 higher3510 Evo higher10BacBench LICENSE file at commit 64b7d45BacBench LICENSE file at commit 64b7d4510 DNABERT-2 higher10
DNABERT-2 higherTieEvo higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (3)
PaperRoleRanks DNABERT-2 higherRanks Evo higherTies
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOIDisagrees060
GENEB leaderboard (leaderboard.json, 13 task-type aggregates) · RepositoryAgrees35100
BacBench LICENSE file at commit 64b7d45 and 12 more records · DOIAgrees1000
Comparisons in Zero-shot benchmarking of RNA language… (6)
Comparisons in GENEB leaderboard (45)
ProtocolDatasetMetricDNABERT-2EvoHigher
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Chromatin Acc.'), unitless0.320.289DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Coding/NC'), unitless0.5820.539DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'DNA Methyl.'), unitless0.04140.0458Evo
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Enhancers'), unitless0.3320.288DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Histone Mod.'), unitless0.2650.258DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Mouse Enh.'), unitless0.09820.12Evo
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Promoters'), unitless0.4710.436DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Regulatory'), unitless0.2160.125DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Species Clf.'), unitless0.4020.207DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Splice Sites'), unitless0.2180.0657DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'TF Binding'), unitless0.1470.0386DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Virus/Phage'), unitless0.1340.0941DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'lncRNA'), unitless0.06270.0189DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; macro average: unweighted mean of the 13 category MCCs), unitless0.2530.194DNABERT-2
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; micro average: mean over all 100 tasks), unitless0.2720.232DNABERT-2
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Chromatin Acc.'), unitless0.1560.14DNABERT-2
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Coding/NC'), unitless0.1620.293Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'DNA Methyl.'), unitless0.002870.0208Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Enhancers'), unitless0.1340.165Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Histone Mod.'), unitless0.1120.108DNABERT-2
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Mouse Enh.'), unitless0.01880.0568Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Promoters'), unitless0.240.279Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Regulatory'), unitless0.07090.0788Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Species Clf.'), unitless0.1770.103DNABERT-2
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Splice Sites'), unitless0.07890.0643DNABERT-2
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'TF Binding'), unitless0.06630.0084DNABERT-2
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Virus/Phage'), unitless0.0421-0.00343DNABERT-2
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'lncRNA'), unitless0.01630.00964DNABERT-2
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; macro average: unweighted mean of the 13 category MCCs), unitless0.09820.102Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; micro average: mean over all 100 tasks), unitless0.1190.126Evo
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Chromatin Acc.'), unitless0.570.284DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Coding/NC'), unitless0.820.719DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'DNA Methyl.'), unitless0.180.0732DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Enhancers'), unitless0.4830.37DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Histone Mod.'), unitless0.4690.397DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Mouse Enh.'), unitless0.5270.287DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Promoters'), unitless0.7080.537DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Regulatory'), unitless0.4350.269DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Species Clf.'), unitless0.6250.285DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Splice Sites'), unitless0.4910.16DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'TF Binding'), unitless0.5740.173DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Virus/Phage'), unitless0.4920.205DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'lncRNA'), unitless0.2650.119DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; macro average: unweighted mean of the 13 category MCCs), unitless0.5110.298DNABERT-2
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; micro average: mean over all 100 tasks), unitless0.5060.342DNABERT-2
Comparisons in BacBench LICENSE file at commit 64b7d45 (10)
ProtocolDatasetMetricDNABERT-2EvoHigher
BacBench antibiotic resistance, binary linear probe AUPRCAntibiotic resistance binary prediction subset, 37 antibioticsAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless52.444.1DNABERT-2
BacBench antibiotic resistance, binary linear probe AUROCAntibiotic resistance binary prediction subset, 37 antibioticsAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless79.968.2DNABERT-2
BacBench antibiotic resistance, MIC regression Pearson correlationAntibiotic resistance MIC regression subset, 56 antibioticsPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless45.929.2DNABERT-2
BacBench antibiotic resistance, MIC regression R squaredAntibiotic resistance MIC regression subset, 56 antibioticsR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless23.610DNABERT-2
BacBench essential gene prediction, linear probe AUPRCBacBench essential gene dataset from the Database of Essential GenesAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless30.213.7DNABERT-2
BacBench essential gene prediction, linear probe AUROCBacBench essential gene dataset from the Database of Essential GenesAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless6551.8DNABERT-2
BacBench phenotypic trait prediction, linear probe AUPRCBacBench phenotypic trait datasetAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless41.939.5DNABERT-2
BacBench phenotypic trait prediction, linear probe AUROCBacBench phenotypic trait datasetAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless63.860.2DNABERT-2
BacBench protein-protein interaction, linear probe AUPRCBacBench protein-protein interaction dataset from STRING DBAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless28.320.9DNABERT-2
BacBench protein-protein interaction, linear probe AUROCBacBench protein-protein interaction dataset from STRING DBAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless61.656.5DNABERT-2

What disagrees

Other studies
rank DNABERT-2 above Evo in 45 of 55 comparisons
This study
ranks Evo above DNABERT-2 in 6 of 6 comparisons
Compared
DNABERT-2, Evo
Study that disagrees
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning, Zero-shot benchmarking of RNA language models: Supplementary Information
Other studies
14 studiesBacBench LICENSE file at commit 64b7d45, BacBench repository README at commit 64b7d45, Bacformer LICENSE file at commit 61dca95, Bacformer repository README at commit 61dca95, Hugging Face model card for macwiatrak/bacformer-causal-complete-genomes, Hugging Face model card for macwiatrak/bacformer-causal-MAG, Hugging Face model card for macwiatrak/bacformer-for-ppi-prediction, Hugging Face model card for macwiatrak/bacformer-large-masked-complete-genomes, Hugging Face model card for macwiatrak/bacformer-large-masked-MAG, Hugging Face model card for macwiatrak/bacformer-masked-complete-genomes, Hugging Face model card for macwiatrak/bacformer-masked-MAG, Bacformer preprint, bioRxiv 2025.07.20.665723 version 2, Bacformer preprint, bioRxiv 2025.07.20.665723 version 3, GENEB leaderboard (leaderboard.json, 13 task-type aggregates)
Rule
Ordering reversal; different-task
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (12)
StudyToolProtocol or benchmarkMetricValue
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningDNABERT-2 (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)AUROC (PLL-D; mean over assays)0.543±0.031
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningDNABERT-2 (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)AUROC (PLL-D; median over assays)0.542
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningDNABERT-2 (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)MCC (PLL-D; mean over assays)0.069±0.052
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningDNABERT-2 (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)MCC (PLL-D; median over assays)0.061
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningDNABERT-2 (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)Absolute spearman correlation (PLL-D; mean over assays)0.085±0.056
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningDNABERT-2 (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)Absolute spearman correlation (PLL-D; median over assays)0.086
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningEVO1.5 (8k_base) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (mean over assays)AUROC (PLL-D; mean over assays)0.588±0.056
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningEVO1.5 (8k_base) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, AUC (median over assays)AUROC (PLL-D; median over assays)0.586
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningEVO1.5 (8k_base) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (mean over assays)MCC (PLL-D; mean over assays)0.132±0.088
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningEVO1.5 (8k_base) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, MCC (median over assays)MCC (PLL-D; median over assays)0.13
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningEVO1.5 (8k_base) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (mean over assays)Absolute spearman correlation (PLL-D; mean over assays)0.179±0.114
Zero-shot benchmarking of RNA language models: Supplementary Information; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningEVO1.5 (8k_base) (zero-shot, Wang et al. 2026)Wang et al. 2026 zero-shot fitness, RNAGym ncRNA DMS assays (31 assays), PLL-D, SR (median over assays)Absolute spearman correlation (PLL-D; median over assays)0.178
In the other studies (110 results)
In the other studies
StudyToolProtocol or benchmarkMetricValue
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUPRCAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)52.35 ± 23.55
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUROCAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)79.88 ± 7.26
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression Pearson correlationPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)45.89 ± 19.27
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression R squaredR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)23.61 ± 18.33
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUPRCAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)44.06 ± 23.28
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUROCAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)68.22 ± 9.32
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression Pearson correlationPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)29.18 ± 17.26
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression R squaredR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)10.01 ± 10.06
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUPRCAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)30.21 ± 19.81
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUROCAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)64.97 ± 8.29
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUPRCAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)13.67 ± 22.40
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUROCAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)51.78 ± 4.16
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUPRCAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)41.93 ± 34.31
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUROCAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)63.81 ± 11.15
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUPRCAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)39.55 ± 33.36
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUROCAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)60.19 ± 11.04
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUPRCAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)28.30 ± 9.55
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUROCAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)61.64 ± 3.58
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUPRCAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)20.93 ± 8.80
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUROCAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)56.45 ± 3.14
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Chromatin Acc.')0.32
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Coding/NC')0.582
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'DNA Methyl.')0.0414
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Enhancers')0.332
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Histone Mod.')0.265
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'lncRNA')0.0627
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; macro average: unweighted mean of the 13 category MCCs)0.253
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; micro average: mean over all 100 tasks)0.272
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Mouse Enh.')0.0982
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Promoters')0.471
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Regulatory')0.216
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Species Clf.')0.402
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Splice Sites')0.218
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'TF Binding')0.147
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Virus/Phage')0.134
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Chromatin Acc.')0.156
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Coding/NC')0.162
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'DNA Methyl.')0.00287
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Enhancers')0.134
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Histone Mod.')0.112
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'lncRNA')0.0163
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; macro average: unweighted mean of the 13 category MCCs)0.0982
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; micro average: mean over all 100 tasks)0.119
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Mouse Enh.')0.0188
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Promoters')0.24
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Regulatory')0.0709
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Species Clf.')0.177
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Splice Sites')0.0789
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'TF Binding')0.0663
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Virus/Phage')0.0421
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Chromatin Acc.')0.57
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Coding/NC')0.82
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'DNA Methyl.')0.18
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Enhancers')0.483
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Histone Mod.')0.469
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'lncRNA')0.265
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; macro average: unweighted mean of the 13 category MCCs)0.511
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; micro average: mean over all 100 tasks)0.506
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Mouse Enh.')0.527
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Promoters')0.708
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Regulatory')0.435
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Species Clf.')0.625
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Splice Sites')0.491
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'TF Binding')0.574
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)dnabert2 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Virus/Phage')0.492
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Chromatin Acc.')0.289
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Coding/NC')0.539
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'DNA Methyl.')0.0458
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Enhancers')0.288
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Histone Mod.')0.258
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'lncRNA')0.0189
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; macro average: unweighted mean of the 13 category MCCs)0.194
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; micro average: mean over all 100 tasks)0.232
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Mouse Enh.')0.12
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Promoters')0.436
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Regulatory')0.125
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Species Clf.')0.207
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Splice Sites')0.0657
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'TF Binding')0.0386
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Virus/Phage')0.0941
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Chromatin Acc.')0.14
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Coding/NC')0.293
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'DNA Methyl.')0.0208
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Enhancers')0.165
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Histone Mod.')0.108
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'lncRNA')0.00964
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; macro average: unweighted mean of the 13 category MCCs)0.102
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; micro average: mean over all 100 tasks)0.126
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Mouse Enh.')0.0568
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Promoters')0.279
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Regulatory')0.0788
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Species Clf.')0.103
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Splice Sites')0.0643
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'TF Binding')0.0084
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Virus/Phage')-0.00343
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Chromatin Acc.')0.284
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Coding/NC')0.719
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'DNA Methyl.')0.0732
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Enhancers')0.37
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Histone Mod.')0.397
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'lncRNA')0.119
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; macro average: unweighted mean of the 13 category MCCs)0.298
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; micro average: mean over all 100 tasks)0.342
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Mouse Enh.')0.287
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Promoters')0.537
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Regulatory')0.269
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Species Clf.')0.285
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Splice Sites')0.16
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'TF Binding')0.173
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Virus/Phage')0.205

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-e8e31199036fbc43 · All contested results · How findings are made and followed up