rewirebio.iobenchmarks
Contested result

ESM-2 and Evo 2: Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) and BacBench LICENSE file at commit 64b7d45 disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) ranks Evo 2 above ESM-2 in 28 of 28 comparisons; BacBench LICENSE file at commit 64b7d45 ranks ESM-2 above Evo 2 in 10 of 10 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

3 papers rank ESM-2 above Evo 2 in 12 of their 43 comparisons and Evo 2 higher in 30; 1 is a tie. Of those papers, Genome modelling and design across all… ranks Evo 2 higher on its own (28 of 28).

Papers grouped by which system they rank higher: ESM-2 or Evo 2Rank ESM-2 higher: BacBench LICENSE file at commit 64b7d45. Rank Evo 2 higher: Genome modelling and design across all…. Mixed: Genomic heterogeneity inflates….Rank ESM-2 higherMixedRank Evo 2 higherBacBench LICENSE fileat commit 64b7d45BacBench LICENSE file at commit 64b7d45Genomic heterogeneityinflates…Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Genome modelling anddesign across all…Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026)
  • Rank ESM-2 higher (1): BacBench LICENSE file at commit 64b7d45
  • Rank Evo 2 higher (1): Genome modelling and design across all…
  • Mixed (1): Genomic heterogeneity inflates…

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores ESM-2 or Evo 2 higherGenome modelling and design across all…: ESM-2 higher in 0, Evo 2 higher in 28, 0 ties. BacBench LICENSE file at commit 64b7d45: ESM-2 higher in 10, Evo 2 higher in 0, 0 ties. Genomic heterogeneity inflates…: ESM-2 higher in 2, Evo 2 higher in 2, 1 tie.Genome modelling and design across all…Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026)28 Evo 2 higher28BacBench LICENSE file at commit 64b7d45BacBench LICENSE file at commit 64b7d4510 ESM-2 higher10Genomic heterogeneity inflates…Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)2 ESM-2 higher21 tie12 Evo 2 higher2
ESM-2 higherTieEvo 2 higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Genome modelling and design across all… (28)
ProtocolDatasetMetricESM-2Evo 2Higher
Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding indels), unitless0.7720.952Evo 2
Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding indels), unitless0.5330.904Evo 2
Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding microsatellite variants), unitless0.2530.812Evo 2
Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding microsatellite variants), unitless0.5420.842Evo 2
Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, far from splice site), unitless0.7410.948Evo 2
Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, far from splice site), unitless0.5310.911Evo 2
Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, highly conserved), unitless0.9460.974Evo 2
Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, highly conserved), unitless0.510.79Evo 2
Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, moderately conserved), unitless0.7350.914Evo 2
Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, moderately conserved), unitless0.5020.813Evo 2
Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, not conserved), unitless0.7580.835Evo 2
Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, not conserved), unitless0.6330.757Evo 2
Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, poorly conserved), unitless0.6520.889Evo 2
Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, poorly conserved), unitless0.550.82Evo 2
Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs), unitless0.8270.889Evo 2
Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs), unitless0.6770.841Evo 2
Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, far from splice site), unitless0.8220.885Evo 2
Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, far from splice site), unitless0.6780.841Evo 2
Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, highly conserved), unitless0.8950.918Evo 2
Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, highly conserved), unitless0.6570.776Evo 2
Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, moderately conserved), unitless0.620.667Evo 2
Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, moderately conserved), unitless0.7060.83Evo 2
Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, near splice site), unitless0.8960.931Evo 2
Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, near splice site), unitless0.680.827Evo 2
Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, not conserved), unitless0.4930.52Evo 2
Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, not conserved), unitless0.7180.868Evo 2
Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, poorly conserved), unitless0.5030.602Evo 2
Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, poorly conserved), unitless0.6960.86Evo 2
Comparisons in BacBench LICENSE file at commit 64b7d45 (10)
ProtocolDatasetMetricESM-2Evo 2Higher
BacBench antibiotic resistance, binary linear probe AUPRCAntibiotic resistance binary prediction subset, 37 antibioticsAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless6238.4ESM-2
BacBench antibiotic resistance, binary linear probe AUROCAntibiotic resistance binary prediction subset, 37 antibioticsAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless8566.3ESM-2
BacBench antibiotic resistance, MIC regression Pearson correlationAntibiotic resistance MIC regression subset, 56 antibioticsPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless54.60.53ESM-2
BacBench antibiotic resistance, MIC regression R squaredAntibiotic resistance MIC regression subset, 56 antibioticsR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless31.23.95ESM-2
BacBench essential gene prediction, linear probe AUPRCBacBench essential gene dataset from the Database of Essential GenesAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless4415.4ESM-2
BacBench essential gene prediction, linear probe AUROCBacBench essential gene dataset from the Database of Essential GenesAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless80.456ESM-2
BacBench phenotypic trait prediction, linear probe AUPRCBacBench phenotypic trait datasetAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless45.425ESM-2
BacBench phenotypic trait prediction, linear probe AUROCBacBench phenotypic trait datasetAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless68.755.4ESM-2
BacBench protein-protein interaction, linear probe AUPRCBacBench protein-protein interaction dataset from STRING DBAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless44.216ESM-2
BacBench protein-protein interaction, linear probe AUROCBacBench protein-protein interaction dataset from STRING DBAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless72.850ESM-2
Comparisons in Genomic heterogeneity inflates… (5)

What disagrees

studies
2
Compared
ESM-2, Evo 2
Study that disagrees
Other studies
Rule
Two-study conflict; different-task
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-ca0f3ede62449d48 · All contested results · How findings are made and followed up