ESM-2 and Evo 2: Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) and BacBench LICENSE file at commit 64b7d45 disagree
- Low severity
- Not yet reviewed
- Two-study conflict
- different-task
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) ranks Evo 2 above ESM-2 in 28 of 28 comparisons; BacBench LICENSE file at commit 64b7d45 ranks ESM-2 above Evo 2 in 10 of 10 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
3 papers rank ESM-2 above Evo 2 in 12 of their 43 comparisons and Evo 2 higher in 30; 1 is a tie. Of those papers, Genome modelling and design across all… ranks Evo 2 higher on its own (28 of 28).
- Rank ESM-2 higher (1): BacBench LICENSE file at commit 64b7d45
- Rank Evo 2 higher (1): Genome modelling and design across all…
- Mixed (1): Genomic heterogeneity inflates…
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks ESM-2 higher | Ranks Evo 2 higher | Ties |
|---|---|---|---|---|
| Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) and 1 more record · DOI | Agrees | 0 | 28 | 0 |
| BacBench LICENSE file at commit 64b7d45 and 12 more records · DOI | Agrees | 10 | 0 | 0 |
| Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) · DOI | Agrees | 2 | 2 | 1 |
Comparisons in Genome modelling and design across all… (28)
Comparisons in BacBench LICENSE file at commit 64b7d45 (10)
| Protocol | Dataset | Metric | ESM-2 | Evo 2 | Higher |
|---|---|---|---|---|---|
| BacBench antibiotic resistance, binary linear probe AUPRC | Antibiotic resistance binary prediction subset, 37 antibiotics | AUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless | 62 | 38.4 | ESM-2 |
| BacBench antibiotic resistance, binary linear probe AUROC | Antibiotic resistance binary prediction subset, 37 antibiotics | AUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless | 85 | 66.3 | ESM-2 |
| BacBench antibiotic resistance, MIC regression Pearson correlation | Antibiotic resistance MIC regression subset, 56 antibiotics | Pearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless | 54.6 | 0.53 | ESM-2 |
| BacBench antibiotic resistance, MIC regression R squared | Antibiotic resistance MIC regression subset, 56 antibiotics | R² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless | 31.2 | 3.95 | ESM-2 |
| BacBench essential gene prediction, linear probe AUPRC | BacBench essential gene dataset from the Database of Essential Genes | AUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless | 44 | 15.4 | ESM-2 |
| BacBench essential gene prediction, linear probe AUROC | BacBench essential gene dataset from the Database of Essential Genes | AUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless | 80.4 | 56 | ESM-2 |
| BacBench phenotypic trait prediction, linear probe AUPRC | BacBench phenotypic trait dataset | AUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless | 45.4 | 25 | ESM-2 |
| BacBench phenotypic trait prediction, linear probe AUROC | BacBench phenotypic trait dataset | AUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless | 68.7 | 55.4 | ESM-2 |
| BacBench protein-protein interaction, linear probe AUPRC | BacBench protein-protein interaction dataset from STRING DB | AUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless | 44.2 | 16 | ESM-2 |
| BacBench protein-protein interaction, linear probe AUROC | BacBench protein-protein interaction dataset from STRING DB | AUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless | 72.8 | 50 | ESM-2 |
Comparisons in Genomic heterogeneity inflates… (5)
What disagrees
- studies
- 2
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-ca0f3ede62449d48 · All contested results · How findings are made and followed up