DNABERT-2 and Evo 2: BacBench LICENSE file at commit 64b7d45 reverses the order
- Low severity
- Not yet reviewed
- Ordering reversal
- unknown
BacBench LICENSE file at commit 64b7d45 ranks DNABERT-2 above Evo 2 in 10 of 10 comparisons; 3 of 3 other studies rank Evo 2 above DNABERT-2.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
4 papers rank Evo 2 above DNABERT-2 in 63 of their 80 comparisons and DNABERT-2 higher in 14; 3 are ties. BacBench LICENSE file at commit 64b7d45 ranks DNABERT-2 above Evo 2 in all 10 of its comparisons.
- Rank Evo 2 higher (4): GFMBench-API, Genomic heterogeneity inflates…, mRNABench, GUANinE v1.1 reveals complementarity…
- Rank DNABERT-2 higher (1): BacBench LICENSE file at commit 64b7d45 (disagrees)
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks Evo 2 higher | Ranks DNABERT-2 higher | Ties |
|---|---|---|---|---|
| BacBench LICENSE file at commit 64b7d45 and 12 more records · DOI | Disagrees | 0 | 10 | 0 |
| GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models · DOI | Agrees | 35 | 13 | 2 |
| Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) · DOI | Agrees | 14 | 0 | 0 |
| mRNABench: A curated benchmark for mature mRNA property and function prediction and 1 more record · DOI | Agrees | 9 | 1 | 0 |
| GUANinE v1.1 reveals complementarity of supervised and genomic language models · DOI | Agrees | 5 | 0 | 1 |
Comparisons in BacBench LICENSE file at commit 64b7d45 (10)
| Protocol | Dataset | Metric | Evo 2 | DNABERT-2 | Higher |
|---|---|---|---|---|---|
| BacBench antibiotic resistance, binary linear probe AUPRC | Antibiotic resistance binary prediction subset, 37 antibiotics | AUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless | 38.4 | 52.4 | DNABERT-2 |
| BacBench antibiotic resistance, binary linear probe AUROC | Antibiotic resistance binary prediction subset, 37 antibiotics | AUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless | 66.3 | 79.9 | DNABERT-2 |
| BacBench antibiotic resistance, MIC regression Pearson correlation | Antibiotic resistance MIC regression subset, 56 antibiotics | Pearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless | 0.53 | 45.9 | DNABERT-2 |
| BacBench antibiotic resistance, MIC regression R squared | Antibiotic resistance MIC regression subset, 56 antibiotics | R² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless | 3.95 | 23.6 | DNABERT-2 |
| BacBench essential gene prediction, linear probe AUPRC | BacBench essential gene dataset from the Database of Essential Genes | AUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless | 15.4 | 30.2 | DNABERT-2 |
| BacBench essential gene prediction, linear probe AUROC | BacBench essential gene dataset from the Database of Essential Genes | AUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless | 56 | 65 | DNABERT-2 |
| BacBench phenotypic trait prediction, linear probe AUPRC | BacBench phenotypic trait dataset | AUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless | 25 | 41.9 | DNABERT-2 |
| BacBench phenotypic trait prediction, linear probe AUROC | BacBench phenotypic trait dataset | AUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless | 55.4 | 63.8 | DNABERT-2 |
| BacBench protein-protein interaction, linear probe AUPRC | BacBench protein-protein interaction dataset from STRING DB | AUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless | 16 | 28.3 | DNABERT-2 |
| BacBench protein-protein interaction, linear probe AUROC | BacBench protein-protein interaction dataset from STRING DB | AUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless | 50 | 61.6 | DNABERT-2 |
Comparisons in GFMBench-API (50)
Comparisons in Genomic heterogeneity inflates… (14)
Comparisons in mRNABench (10)
Comparisons in GUANinE v1.1 reveals complementarity… (6)
| Protocol | Dataset | Metric | Evo 2 | DNABERT-2 | Higher |
|---|---|---|---|---|---|
| GUANinE v1.1 ccre-propensity, Spearman rho x100 (Table 2) | GUANinE ccre-propensity test set | Spearman (GUANinE v1.1 test set), unitless | 40.2 | 32.8 | Evo 2 |
| GUANinE v1.1 cons100, Spearman rho x100 (Table 2) | GUANinE cons100 test set | Spearman (GUANinE v1.1 test set), unitless | 55.2 | 50.4 | Evo 2 |
| GUANinE v1.1 cons30, Spearman rho x100 (Table 2) | GUANinE cons30 test set | Spearman (GUANinE v1.1 test set), unitless | 68.5 | 60.6 | Evo 2 |
| GUANinE v1.1 dnase-propensity, Spearman rho x100 (Table 2) | GUANinE dnase-propensity test set | Spearman (GUANinE v1.1 test set), unitless | 41.8 | 41.7 | Tie |
| GUANinE v1.1 gpra-c, Spearman rho x100 (Table 2) | GUANinE gpra-c test set | Spearman (GUANinE v1.1 test set), unitless | 72.7 | 69.9 | Evo 2 |
| GUANinE v1.1 gpra-d, Spearman rho x100 (Table 2) | GUANinE gpra-d test set | Spearman (GUANinE v1.1 test set), unitless | 76.4 | 73.9 | Evo 2 |
What disagrees
- Other studies
- rank Evo 2 above DNABERT-2 in 28 of 29 comparisons
- This study
- ranks DNABERT-2 above Evo 2 in 10 of 10 comparisons
- Compared
- DNABERT-2, Evo 2
- Study that disagrees
- BacBench LICENSE file at commit 64b7d45, BacBench repository README at commit 64b7d45, Bacformer LICENSE file at commit 61dca95, Bacformer repository README at commit 61dca95, Hugging Face model card for macwiatrak/bacformer-causal-complete-genomes, Hugging Face model card for macwiatrak/bacformer-causal-MAG, Hugging Face model card for macwiatrak/bacformer-for-ppi-prediction, Hugging Face model card for macwiatrak/bacformer-large-masked-complete-genomes, Hugging Face model card for macwiatrak/bacformer-large-masked-MAG, Hugging Face model card for macwiatrak/bacformer-masked-complete-genomes, Hugging Face model card for macwiatrak/bacformer-masked-MAG, Bacformer preprint, bioRxiv 2025.07.20.665723 version 2, Bacformer preprint, bioRxiv 2025.07.20.665723 version 3
- Other studies
- mRNABench: A curated benchmark for mature mRNA property and function prediction, GUANinE v1.1 reveals complementarity of supervised and genomic language models, Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2), mRNABench: A curated benchmark for mature mRNA property and function prediction
- Rule
- Ordering reversal; unknown
- Model identity
- Matched across studies partly through links not yet reviewed
The compared numbers
Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.
In the other studies (58 results)
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-e3c4da79628e000d · All contested results · How findings are made and followed up