rewirebio.iobenchmarks
Contested result

DNABERT-2 and Evo 2: BacBench LICENSE file at commit 64b7d45 reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

BacBench LICENSE file at commit 64b7d45 ranks DNABERT-2 above Evo 2 in 10 of 10 comparisons; 3 of 3 other studies rank Evo 2 above DNABERT-2.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

4 papers rank Evo 2 above DNABERT-2 in 63 of their 80 comparisons and DNABERT-2 higher in 14; 3 are ties. BacBench LICENSE file at commit 64b7d45 ranks DNABERT-2 above Evo 2 in all 10 of its comparisons.

Papers grouped by which system they rank higher: Evo 2 or DNABERT-2Rank Evo 2 higher: GFMBench-API, Genomic heterogeneity inflates…, mRNABench, GUANinE v1.1 reveals complementarity…. Rank DNABERT-2 higher: BacBench LICENSE file at commit 64b7d45 (disagrees). Mixed: none.Rank Evo 2 higherMixedRank DNABERT-2 higherGFMBench-APIGFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation ModelsGenomic heterogeneityinflates…Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)mRNABenchmRNABench: A curated benchmark for mature mRNA property and function predictionGUANinE v1.1 revealscomplementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language modelsBacBench LICENSE fileat commit 64b7d45BacBench LICENSE file at commit 64b7d45
  • Rank Evo 2 higher (4): GFMBench-API, Genomic heterogeneity inflates…, mRNABench, GUANinE v1.1 reveals complementarity…
  • Rank DNABERT-2 higher (1): BacBench LICENSE file at commit 64b7d45 (disagrees)
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Evo 2 or DNABERT-2 higherBacBench LICENSE file at commit 64b7d45 (disagrees): Evo 2 higher in 0, DNABERT-2 higher in 10, 0 ties. GFMBench-API: Evo 2 higher in 35, DNABERT-2 higher in 13, 2 ties. Genomic heterogeneity inflates…: Evo 2 higher in 14, DNABERT-2 higher in 0, 0 ties. mRNABench: Evo 2 higher in 9, DNABERT-2 higher in 1, 0 ties. GUANinE v1.1 reveals complementarity…: Evo 2 higher in 5, DNABERT-2 higher in 0, 1 tie.Disagrees: BacBench LICENSE file at commit 64b7d45BacBench LICENSE file at commit 64b7d4510 DNABERT-2 higher10GFMBench-APIGFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models35 Evo 2 higher352 tie13 DNABERT-2 higher13Genomic heterogeneity inflates…Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)14 Evo 2 higher14mRNABenchmRNABench: A curated benchmark for mature mRNA property and function prediction9 Evo 2 higher91 DNABERT-2 higher1GUANinE v1.1 reveals complementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language models5 Evo 2 higher51 tie1
Evo 2 higherTieDNABERT-2 higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in BacBench LICENSE file at commit 64b7d45 (10)
ProtocolDatasetMetricEvo 2DNABERT-2Higher
BacBench antibiotic resistance, binary linear probe AUPRCAntibiotic resistance binary prediction subset, 37 antibioticsAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless38.452.4DNABERT-2
BacBench antibiotic resistance, binary linear probe AUROCAntibiotic resistance binary prediction subset, 37 antibioticsAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless66.379.9DNABERT-2
BacBench antibiotic resistance, MIC regression Pearson correlationAntibiotic resistance MIC regression subset, 56 antibioticsPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless0.5345.9DNABERT-2
BacBench antibiotic resistance, MIC regression R squaredAntibiotic resistance MIC regression subset, 56 antibioticsR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless3.9523.6DNABERT-2
BacBench essential gene prediction, linear probe AUPRCBacBench essential gene dataset from the Database of Essential GenesAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless15.430.2DNABERT-2
BacBench essential gene prediction, linear probe AUROCBacBench essential gene dataset from the Database of Essential GenesAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless5665DNABERT-2
BacBench phenotypic trait prediction, linear probe AUPRCBacBench phenotypic trait datasetAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless2541.9DNABERT-2
BacBench phenotypic trait prediction, linear probe AUROCBacBench phenotypic trait datasetAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless55.463.8DNABERT-2
BacBench protein-protein interaction, linear probe AUPRCBacBench protein-protein interaction dataset from STRING DBAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless1628.3DNABERT-2
BacBench protein-protein interaction, linear probe AUROCBacBench protein-protein interaction dataset from STRING DBAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless5061.6DNABERT-2
Comparisons in GFMBench-API (50)
ProtocolDatasetMetricEvo 2DNABERT-2Higher
GFMBench-API Bend Variant Effects Disease, zs-embeddingBend Variant Effects Disease test set (GFMBench-API)AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.5960.149Evo 2
GFMBench-API Bend Variant Effects Disease, zs-embeddingBend Variant Effects Disease test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.09440.0751Evo 2
GFMBench-API Bend Variant Effects Disease, zs-embeddingBend Variant Effects Disease test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.4740.152Evo 2
GFMBench-API Bend Variant Effects Disease, zs-embeddingBend Variant Effects Disease test set (GFMBench-API)AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.9120.693Evo 2
GFMBench-API Bend Variant Effects Disease, zs-embeddingBend Variant Effects Disease test set (GFMBench-API)AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.6150.511Evo 2
GFMBench-API Bend Variant Effects Disease, zs-embeddingBend Variant Effects Disease test set (GFMBench-API)AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.8720.696Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embeddingBend Variant Effects Expression test set (GFMBench-API)AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.08970.0853Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embeddingBend Variant Effects Expression test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.08870.0732Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embeddingBend Variant Effects Expression test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.09040.0848Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embeddingBend Variant Effects Expression test set (GFMBench-API)AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.5730.551Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embeddingBend Variant Effects Expression test set (GFMBench-API)AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.5570.487Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embeddingBend Variant Effects Expression test set (GFMBench-API)AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.5820.548Evo 2
GFMBench-API BRCA1, zs-embeddingBRCA1 test set (GFMBench-API)AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.5750.199Evo 2
GFMBench-API BRCA1, zs-embeddingBRCA1 test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.2040.206DNABERT-2
GFMBench-API BRCA1, zs-embeddingBRCA1 test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.540.214Evo 2
GFMBench-API BRCA1, zs-embeddingBRCA1 test set (GFMBench-API)AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.7750.476Evo 2
GFMBench-API BRCA1, zs-embeddingBRCA1 test set (GFMBench-API)AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.4670.49DNABERT-2
GFMBench-API BRCA1, zs-embeddingBRCA1 test set (GFMBench-API)AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.7470.487Evo 2
GFMBench-API ClinVar Indel Pathogenicity, zs-embeddingClinVar Indel Pathogenicity test set (GFMBench-API)AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.9930.862Evo 2
GFMBench-API ClinVar Indel Pathogenicity, zs-embeddingClinVar Indel Pathogenicity test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.9890.86Evo 2
GFMBench-API ClinVar Indel Pathogenicity, zs-embeddingClinVar Indel Pathogenicity test set (GFMBench-API)AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.9670.508Evo 2
GFMBench-API ClinVar Indel Pathogenicity, zs-embeddingClinVar Indel Pathogenicity test set (GFMBench-API)AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.9460.508Evo 2
GFMBench-API LOL-EVE, zs-embeddingLOL-EVE test set (GFMBench-API)AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.09380.0947Tie
GFMBench-API LOL-EVE, zs-embeddingLOL-EVE test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.08450.105DNABERT-2
GFMBench-API LOL-EVE, zs-embeddingLOL-EVE test set (GFMBench-API)AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.5430.579DNABERT-2
GFMBench-API LOL-EVE, zs-embeddingLOL-EVE test set (GFMBench-API)AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.5310.601DNABERT-2
GFMBench-API LRB Variant Effect OMIM, zs-embeddingLRB Variant Effect OMIM test set (GFMBench-API)AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.00210.0022DNABERT-2
GFMBench-API LRB Variant Effect OMIM, zs-embeddingLRB Variant Effect OMIM test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.0020.0019Evo 2
GFMBench-API LRB Variant Effect OMIM, zs-embeddingLRB Variant Effect OMIM test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.0020.0021DNABERT-2
GFMBench-API LRB Variant Effect OMIM, zs-embeddingLRB Variant Effect OMIM test set (GFMBench-API)AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.5020.531DNABERT-2
GFMBench-API LRB Variant Effect OMIM, zs-embeddingLRB Variant Effect OMIM test set (GFMBench-API)AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.4980.505DNABERT-2
GFMBench-API LRB Variant Effect OMIM, zs-embeddingLRB Variant Effect OMIM test set (GFMBench-API)AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.5010.512DNABERT-2
GFMBench-API Songlab ClinVar, zs-embeddingSonglab ClinVar test set (GFMBench-API)AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.8650.547Evo 2
GFMBench-API Songlab ClinVar, zs-embeddingSonglab ClinVar test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.5760.552Evo 2
GFMBench-API Songlab ClinVar, zs-embeddingSonglab ClinVar test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.7340.547Evo 2
GFMBench-API Songlab ClinVar, zs-embeddingSonglab ClinVar test set (GFMBench-API)AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.8450.495Evo 2
GFMBench-API Songlab ClinVar, zs-embeddingSonglab ClinVar test set (GFMBench-API)AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.5790.501Evo 2
GFMBench-API Songlab ClinVar, zs-embeddingSonglab ClinVar test set (GFMBench-API)AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.7220.496Evo 2
GFMBench-API TraitGym Complex, zs-embeddingTraitGym Complex test set (GFMBench-API)AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.1070.112DNABERT-2
GFMBench-API TraitGym Complex, zs-embeddingTraitGym Complex test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.1060.0986Evo 2
GFMBench-API TraitGym Complex, zs-embeddingTraitGym Complex test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.1080.109DNABERT-2
GFMBench-API TraitGym Complex, zs-embeddingTraitGym Complex test set (GFMBench-API)AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.5240.536DNABERT-2
GFMBench-API TraitGym Complex, zs-embeddingTraitGym Complex test set (GFMBench-API)AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.5230.498Evo 2
GFMBench-API TraitGym Complex, zs-embeddingTraitGym Complex test set (GFMBench-API)AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.5270.53Tie
GFMBench-API TraitGym Mendelian, zs-embeddingTraitGym Mendelian test set (GFMBench-API)AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.170.153Evo 2
GFMBench-API TraitGym Mendelian, zs-embeddingTraitGym Mendelian test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.1210.089Evo 2
GFMBench-API TraitGym Mendelian, zs-embeddingTraitGym Mendelian test set (GFMBench-API)AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.1640.147Evo 2
GFMBench-API TraitGym Mendelian, zs-embeddingTraitGym Mendelian test set (GFMBench-API)AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless0.6260.605Evo 2
GFMBench-API TraitGym Mendelian, zs-embeddingTraitGym Mendelian test set (GFMBench-API)AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless0.5480.463Evo 2
GFMBench-API TraitGym Mendelian, zs-embeddingTraitGym Mendelian test set (GFMBench-API)AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless0.6180.594Evo 2
Comparisons in Genomic heterogeneity inflates… (14)
ProtocolDatasetMetricEvo 2DNABERT-2Higher
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: All variants (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, All variants), unitless0.9690.576Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Intron (Non-splice) (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Intron (Non-splice)), unitless0.9030.437Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Missense (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Missense), unitless0.8360.484Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Missense & Intron (Non-splice) (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Missense & Intron (Non-splice)), unitless0.8350.449Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Missense & 3'UTR (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Missense & 3'UTR), unitless0.8960.546Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Noncoding (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Noncoding), unitless0.9750.433Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: RNA gene (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, RNA gene), unitless0.4590.423Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Splice (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Splice), unitless0.6970.454Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Start loss (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Start loss), unitless0.7110.389Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Stop gain (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Stop gain), unitless0.6630.483Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Stop loss (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Stop loss), unitless0.8260.57Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Synonymous (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Synonymous), unitless0.8850.552Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: 3' UTR (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, 3' UTR), unitless0.7470.499Evo 2
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: 5' UTR (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, 5' UTR), unitless0.7670.414Evo 2
Comparisons in mRNABench (10)
Comparisons in GUANinE v1.1 reveals complementarity… (6)

What disagrees

Other studies
rank Evo 2 above DNABERT-2 in 28 of 29 comparisons
This study
ranks DNABERT-2 above Evo 2 in 10 of 10 comparisons
Compared
DNABERT-2, Evo 2
Study that disagrees
BacBench LICENSE file at commit 64b7d45, BacBench repository README at commit 64b7d45, Bacformer LICENSE file at commit 61dca95, Bacformer repository README at commit 61dca95, Hugging Face model card for macwiatrak/bacformer-causal-complete-genomes, Hugging Face model card for macwiatrak/bacformer-causal-MAG, Hugging Face model card for macwiatrak/bacformer-for-ppi-prediction, Hugging Face model card for macwiatrak/bacformer-large-masked-complete-genomes, Hugging Face model card for macwiatrak/bacformer-large-masked-MAG, Hugging Face model card for macwiatrak/bacformer-masked-complete-genomes, Hugging Face model card for macwiatrak/bacformer-masked-MAG, Bacformer preprint, bioRxiv 2025.07.20.665723 version 2, Bacformer preprint, bioRxiv 2025.07.20.665723 version 3
Other studies
mRNABench: A curated benchmark for mature mRNA property and function prediction, GUANinE v1.1 reveals complementarity of supervised and genomic language models, Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2), mRNABench: A curated benchmark for mature mRNA property and function prediction
Rule
Ordering reversal; unknown
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (20)
StudyToolProtocol or benchmarkMetricValue
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUPRCAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)52.35 ± 23.55
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUROCAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)79.88 ± 7.26
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression Pearson correlationPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)45.89 ± 19.27
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression R squaredR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)23.61 ± 18.33
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUPRCAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)38.35 ± 23.08
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUROCAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)66.27 ± 10.19
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression Pearson correlationPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)00.53 ± 00.72
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo-2 (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression R squaredR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)03.95 ± 21.04
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUPRCAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)30.21 ± 19.81
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUROCAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)64.97 ± 8.29
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo-2 (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUPRCAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)15.37 ± 23.71
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo-2 (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUROCAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)56.05 ± 9.46
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUPRCAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)41.93 ± 34.31
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUROCAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)63.81 ± 11.15
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo-2 (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUPRCAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)24.96 ± 20.52
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo-2 (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUROCAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)55.41 ± 8.85
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUPRCAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)28.30 ± 9.55
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3DNABERT-2 (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUROCAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)61.64 ± 3.58
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo-2 (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUPRCAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)15.98 ± 7.37
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo-2 (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUROCAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)50.00 ± 0.00
In the other studies (58 results)
In the other studies
StudyToolProtocol or benchmarkMetricValue
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsDNABERT-2 (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 ccre-propensity, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)32.8
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsDNABERT-2 (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 cons100, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)50.4
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsDNABERT-2 (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 cons30, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)60.6
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsDNABERT-2 (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 gpra-c, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)69.9
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsDNABERT-2 (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 gpra-d, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)73.9
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsEvo2_1B_base (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 ccre-propensity, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)40.2
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsEvo2_1B_base (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 cons100, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)55.2
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsEvo2_1B_base (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 cons30, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)68.5
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsEvo2_1B_base (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 gpra-c, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)72.7
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsEvo2_1B_base (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 gpra-d, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)76.4
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, All variants)0.576
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, All variants)0.969
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Intron (Non-splice))0.437
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Intron (Non-splice))0.903
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Missense & Intron (Non-splice))0.449
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Missense & Intron (Non-splice))0.835
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Missense & 3'UTR)0.546
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Missense & 3'UTR)0.896
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Missense)0.484
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Missense)0.836
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Noncoding)0.433
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Noncoding)0.975
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, RNA gene)0.423
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, RNA gene)0.459
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Splice)0.454
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Splice)0.697
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Start loss)0.389
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Start loss)0.711
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Stop gain)0.483
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Stop gain)0.663
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Stop loss)0.57
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Stop loss)0.826
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Synonymous)0.552
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, Synonymous)0.885
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, 3' UTR)0.499
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, 3' UTR)0.747
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)DNABERT-2 117M, CLS embedding L2 distance (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, 5' UTR)0.414
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Evo 2 7B base, 8,192 bp log-likelihood ratio (Lu et al. 2025)ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)AUROC (ClinVar pathogenic vs benign SNVs, 5' UTR)0.767
mRNABench: A curated benchmark for mature mRNA property and function predictionDNABERT2mRNABench ECLIP: eCLIP binding site predictionAUPRC37.1
mRNABench: A curated benchmark for mature mRNA property and function predictionDNABERT2mRNABench GO: Gene Ontology term predictionAUPRC32.6
mRNABench: A curated benchmark for mature mRNA property and function predictionDNABERT2mRNABench HL: mRNA half lifePearson0.48
mRNABench: A curated benchmark for mature mRNA property and function predictionDNABERT2mRNABench MRL-HL-PAIR: Paired mean ribosome load and half lifePearson0.53
mRNABench: A curated benchmark for mature mRNA property and function predictionDNABERT2mRNABench MRL-MPRA: Mean ribosome load on an MPRA libraryPearson0.57
mRNABench: A curated benchmark for mature mRNA property and function predictionDNABERT2mRNABench MRL: Mean ribosome loadPearson0.29
mRNABench: A curated benchmark for mature mRNA property and function predictionDNABERT2mRNABench MRNA-LOC-LR: mRNA localisation, long rangeAUPRC76.2
mRNABench: A curated benchmark for mature mRNA property and function predictionDNABERT2mRNABench MRNA-LOC-SR: mRNA localisation, short rangeAUPRC67.4
mRNABench: A curated benchmark for mature mRNA property and function predictionDNABERT2mRNABench PROT-LOC: Protein localisationAUPRC32.5
mRNABench: A curated benchmark for mature mRNA property and function predictionDNABERT2mRNABench VEP: Variant effect predictionAUPRC26.4
mRNABench: A curated benchmark for mature mRNA property and function predictionEvo2mRNABench ECLIP: eCLIP binding site predictionAUPRC47.3
mRNABench: A curated benchmark for mature mRNA property and function predictionEvo2mRNABench GO: Gene Ontology term predictionAUPRC46.7
mRNABench: A curated benchmark for mature mRNA property and function predictionEvo2mRNABench HL: mRNA half lifePearson0.66
mRNABench: A curated benchmark for mature mRNA property and function predictionEvo2mRNABench MRL-HL-PAIR: Paired mean ribosome load and half lifePearson0.51
mRNABench: A curated benchmark for mature mRNA property and function predictionEvo2mRNABench MRL-MPRA: Mean ribosome load on an MPRA libraryPearson0.7
mRNABench: A curated benchmark for mature mRNA property and function predictionEvo2mRNABench MRL: Mean ribosome loadPearson0.45
mRNABench: A curated benchmark for mature mRNA property and function predictionEvo2mRNABench MRNA-LOC-LR: mRNA localisation, long rangeAUPRC79.5
mRNABench: A curated benchmark for mature mRNA property and function predictionEvo2mRNABench MRNA-LOC-SR: mRNA localisation, short rangeAUPRC76.7
mRNABench: A curated benchmark for mature mRNA property and function predictionEvo2mRNABench PROT-LOC: Protein localisationAUPRC40.8
mRNABench: A curated benchmark for mature mRNA property and function predictionEvo2mRNABench VEP: Variant effect predictionAUPRC32.1

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-e3c4da79628e000d · All contested results · How findings are made and followed up