ESM-2 and ESM-1b: Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree
- Low severity
- Not yet reviewed
- Two-study conflict
- different-task
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) ranks ESM-1b above ESM-2 in 26 of 28 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-2 above ESM-1b in 6 of 7 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
5 papers rank ESM-2 above ESM-1b in 31 of their 87 comparisons and ESM-1b higher in 47; 9 are ties. Of those papers, Genome modelling and design across all… ranks ESM-1b higher on its own (26 of 28).
- Rank ESM-2 higher (1): Livesey and Marsh 2025
- Rank ESM-1b higher (1): Genome modelling and design across all…
- Mixed (3): HyCell CITATION.cff, ProteinGym, Genomic heterogeneity inflates…
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks ESM-2 higher | Ranks ESM-1b higher | Ties |
|---|---|---|---|---|
| Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) and 1 more record · DOI | Agrees | 2 | 26 | 0 |
| HyCell CITATION.cff (commit 0dcc533c) and 17 more records · Repository | Agrees | 13 | 9 | 6 |
| ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and 6 more records · DOI | Agrees | 8 | 8 | 3 |
| Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and 1 more record · DOI | Agrees | 6 | 1 | 0 |
| Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) · DOI | Agrees | 2 | 3 | 0 |
Comparisons in Genome modelling and design across all… (28)
Comparisons in HyCell CITATION.cff (28)
| Protocol | Dataset | Metric | ESM-2 | ESM-1b | Higher |
|---|---|---|---|---|---|
| AAV low-vs-high (HyPLM report) | FLIP AAV, low-vs-high split (FLIP split) | Spearman (Spearman correlation on the FLIP AAV low-vs-high split), unitless | 0.279 | 0.28 | Tie |
| AAV two-vs-many (HyPLM report) | FLIP AAV, 2-vs-rest split (FLIP split) | Spearman (Spearman correlation on the FLIP AAV two-vs-many split), unitless | 0.518 | 0.544 | ESM-1b |
| Amylase by mutation count (HyPLM report) | Amylase by mutation count (FLIP2, HyPLM report) | Spearman (Spearman correlation on the FLIP2 amylase split partitioned by mutation count), unitless | 0.575 | 0.503 | ESM-2 |
| Amylase one-to-many (HyPLM report) | Amylase one-to-many (FLIP2, HyPLM report) | Spearman (Spearman correlation on the FLIP2 amylase one-to-many split), unitless | 0.64 | 0.653 | ESM-1b |
| Arch retrieval (DGEB) (HyPLM report) | DGEB Arch retrieval (HyPLM report) | Mean average precision at 5 (Zero-shot retrieval mean average precision at 5 on DGEB Arch, ranked by cosine similarity between mean-pooled representations), fraction | 0.12 | 0.206 | ESM-1b |
| Contacts, CASP14 (HyPLM report) | CASP14 evaluation domains (82, HyPLM report) | Precision at l 5 (Long-range contact precision of the top L/5 predictions (sequence separation at least 24) on 82 CASP14 evaluation domains), fraction | 0.366 | 0.322 | ESM-2 |
| Contacts, CASP15 (HyPLM report) | CASP15 evaluation domains (95, HyPLM report) | Precision at l 5 (Long-range contact precision of the top L/5 predictions (sequence separation at least 24) on 95 CASP15 evaluation domains), fraction | 0.391 | 0.404 | ESM-1b |
| Contacts, ProteinNet (HyPLM report) | ProteinNet CASP12 | Precision at l 5 (Long-range contact precision of the top L/5 predictions (sequence separation at least 24) on the ProteinNet CASP12 test set of 40 proteins), fraction | 0.392 | 0.41 | ESM-1b |
| EC classification (DGEB) (HyPLM report) | DGEB EC classification (HyPLM report) | Macro F1 (Macro-averaged F1 of Enzyme Commission classification on DGEB, with the layer and regularisation chosen by 5-fold cross-validation on the training split because this task has no validation split), fraction | 0.665 | 0.673 | ESM-1b |
| EC, DeepFRI split (HyPLM report) | EC annotation, DeepFRI split (HyPLM report) | F1 max (Protein-centric Fmax over Enzyme Commission numbers on the DeepFRI EC split), fraction | 0.874 | 0.878 | Tie |
| EC, SaProt split (HyPLM report) | EC annotation, SaProt split (HyPLM report) | F1 max (Protein-centric Fmax over Enzyme Commission numbers on the SaProt EC split), fraction | 0.888 | 0.889 | Tie |
| Fluorescence (HyPLM report) | Fluorescence (TAPE) | Spearman (Spearman correlation on the TAPE fluorescence regression split), unitless | 0.664 | 0.659 | Tie |
| GB1 low-vs-high (HyPLM report) | FLIP GB1, low-vs-high split (FLIP split) | Spearman (Spearman correlation on the FLIP GB1 low-vs-high split), unitless | 0.377 | 0.417 | ESM-1b |
| GO-BP (HyPLM report) | GO biological process, DeepFRI split via SaProt (HyPLM report) | F1 max (Protein-centric Fmax over Gene Ontology biological-process terms on the DeepFRI GO split as packaged by SaProt), fraction | 0.496 | 0.485 | ESM-2 |
| GO-CC (HyPLM report) | GO cellular component, DeepFRI split via SaProt (HyPLM report) | F1 max (Protein-centric Fmax over Gene Ontology cellular-component terms on the DeepFRI GO split as packaged by SaProt), fraction | 0.535 | 0.52 | ESM-2 |
| GO-MF (HyPLM report) | GO molecular function, DeepFRI split via SaProt (HyPLM report) | F1 max (Protein-centric Fmax over Gene Ontology molecular-function terms on the DeepFRI GO split as packaged by SaProt), fraction | 0.667 | 0.659 | ESM-2 |
| HumanPPI (HyPLM report) | HumanPPI via PEER (HyPLM report) | Accuracy (Human protein-protein interaction classification accuracy on PEER HumanPPI, with pair features [a, b, a*b, |a-b|]), fraction | 0.856 | 0.867 | ESM-1b |
| Localization, 10-class (HyPLM report) | DeepLoc ten-class localisation via PEER (HyPLM report) | Accuracy (Ten-class subcellular localisation accuracy on DeepLoc 1.0 via PEER), fraction | 0.806 | 0.793 | ESM-2 |
| Localization, binary (HyPLM report) | DeepLoc binary localisation via PEER (HyPLM report) | Accuracy (Binary localisation accuracy on DeepLoc 1.0 via PEER), fraction | 0.907 | 0.922 | ESM-1b |
| Meltome human cell (HyPLM report) | FLIP Thermostability, Human-Cell split (FLIP split) | Spearman (Spearman correlation on the FLIP Meltome human-cell split), unitless | 0.696 | 0.698 | Tie |
| Meltome mixed (HyPLM report) | FLIP Thermostability, Mixed split (FLIP split) | Spearman (Spearman correlation on the FLIP Meltome mixed split), unitless | 0.633 | 0.593 | ESM-2 |
| Metal-ion binding (HyPLM report) | Metal-ion binding, SaProt packaging (HyPLM report) | Accuracy (Metal-ion binding classification accuracy on the split packaged by SaProt), fraction | 0.725 | 0.69 | ESM-2 |
| NLL, held-out UniRef panel (HyPLM report) | Held-out UniRef panel (479 proteins, HyPLM report) | Log loss (Mean negative log-likelihood of masked residues in nats on the report's held-out UniRef panel, identical masks and crops for every model), unitless | 1.82 | 1.86 | ESM-2 |
| ProteinGym zero-shot (HyPLM report) | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (Spearman correlation averaged over the 217 substitution assays of ProteinGym v1.3 with the benchmark's official aggregation, from masked-marginal zero-shot scores; per-assay values rounded to three decimals before aggregation), unitless | 0.415 | 0.389 | ESM-2 |
| Remote homology (fold) (HyPLM report) | Fold classification (TAPE) | Accuracy (Fold-level classification accuracy on the TAPE remote-homology (SCOP) test set), fraction | 0.373 | 0.362 | ESM-2 |
| Secondary structure (CB513) (HyPLM report) | Secondary structure prediction (TAPE) | Accuracy (Three-state (Q3) per-residue secondary-structure accuracy on the CB513 test set), fraction | 0.844 | 0.814 | ESM-2 |
| Stability (HyPLM report) | Stability (TAPE, HyPLM report) | Spearman (Spearman correlation on the TAPE stability regression split), unitless | 0.639 | 0.596 | ESM-2 |
| Thermostability (HyPLM report) | Thermostability, Meltome human cell via SaProt (HyPLM report) | Spearman (Spearman correlation on the Meltome human-cell split as packaged by SaProt, which the report lists separately from its own FLIP Meltome human-cell row), unitless | 0.697 | 0.699 | Tie |
Comparisons in ProteinGym (19)
| Protocol | Dataset | Metric | ESM-2 | ESM-1b | Higher |
|---|---|---|---|---|---|
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless | 0.405 | 0.428 | ESM-1b |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless | 0.317 | 0.287 | ESM-2 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless | 0.405 | 0.406 | Tie |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.467 | 0.48 | ESM-1b |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless | 0.432 | 0.435 | Tie |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.341 | 0.365 | ESM-1b |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.422 | 0.396 | ESM-2 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless | 0.387 | 0.349 | ESM-2 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless | 0.467 | 0.491 | ESM-1b |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless | 0.436 | 0.438 | Tie |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless | 0.488 | 0.5 | ESM-1b |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless | 0.333 | 0.26 | ESM-2 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless | 0.4 | 0.394 | ESM-2 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.405 | 0.383 | ESM-2 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.202 | 0.226 | ESM-1b |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.219 | 0.172 | ESM-2 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.173 | 0.15 | ESM-2 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.234 | 0.27 | ESM-1b |
| ProteinGym AMFR official zero-shot Spearman snapshot | ProteinGym AMFR 4G3O official snapshot cohort | Spearman, unitless | 0.26 | 0.374 | ESM-1b |
Comparisons in Livesey and Marsh 2025 (7)
| Protocol | Dataset | Metric | ESM-2 | ESM-1b | Higher |
|---|---|---|---|---|---|
| Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense | Livesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent | 51.4 | 54.3 | ESM-1b |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent | 72.2 | 65 | ESM-2 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent | 27.8 | 14.7 | ESM-2 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent | 74.2 | 62.4 | ESM-2 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent | 70.8 | 67 | ESM-2 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent | 70.5 | 64.5 | ESM-2 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filled | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent | 71.1 | 65.9 | ESM-2 |
Comparisons in Genomic heterogeneity inflates… (5)
What disagrees
- studies
- 2
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-cbcad13670a11473 · All contested results · How findings are made and followed up