rewirebio.iobenchmarks
Contested result

ESM-2 and ESM-1b: Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) ranks ESM-1b above ESM-2 in 26 of 28 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-2 above ESM-1b in 6 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

5 papers rank ESM-2 above ESM-1b in 31 of their 87 comparisons and ESM-1b higher in 47; 9 are ties. Of those papers, Genome modelling and design across all… ranks ESM-1b higher on its own (26 of 28).

Papers grouped by which system they rank higher: ESM-2 or ESM-1bRank ESM-2 higher: Livesey and Marsh 2025. Rank ESM-1b higher: Genome modelling and design across all…. Mixed: HyCell CITATION.cff, ProteinGym, Genomic heterogeneity inflates….Rank ESM-2 higherMixedRank ESM-1b higherLivesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)HyCellCITATION.cffHyCell CITATION.cff (commit 0dcc533c)ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and DesignGenomic heterogeneityinflates…Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)Genome modelling anddesign across all…Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026)
  • Rank ESM-2 higher (1): Livesey and Marsh 2025
  • Rank ESM-1b higher (1): Genome modelling and design across all…
  • Mixed (3): HyCell CITATION.cff, ProteinGym, Genomic heterogeneity inflates…

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores ESM-2 or ESM-1b higherGenome modelling and design across all…: ESM-2 higher in 2, ESM-1b higher in 26, 0 ties. HyCell CITATION.cff: ESM-2 higher in 13, ESM-1b higher in 9, 6 ties. ProteinGym: ESM-2 higher in 8, ESM-1b higher in 8, 3 ties. Livesey and Marsh 2025: ESM-2 higher in 6, ESM-1b higher in 1, 0 ties. Genomic heterogeneity inflates…: ESM-2 higher in 2, ESM-1b higher in 3, 0 ties.Genome modelling and design across all…Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026)2 ESM-2 higher26 ESM-1b higher26HyCell CITATION.cffHyCell CITATION.cff (commit 0dcc533c)13 ESM-2 higher136 tie69 ESM-1b higher9ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design8 ESM-2 higher83 tie38 ESM-1b higher8Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)6 ESM-2 higher61 ESM-1b higher1Genomic heterogeneity inflates…Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)2 ESM-2 higher23 ESM-1b higher3
ESM-2 higherTieESM-1b higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Genome modelling and design across all… (28)
ProtocolDatasetMetricESM-2ESM-1bHigher
Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding indels), unitless0.7720.931ESM-1b
Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding indels), unitless0.5330.809ESM-1b
Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding microsatellite variants), unitless0.2530.692ESM-1b
Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding microsatellite variants), unitless0.5420.814ESM-1b
Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, far from splice site), unitless0.7410.92ESM-1b
Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, far from splice site), unitless0.5310.808ESM-1b
Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, highly conserved), unitless0.9460.986ESM-1b
Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, highly conserved), unitless0.510.832ESM-1b
Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, moderately conserved), unitless0.7350.897ESM-1b
Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, moderately conserved), unitless0.5020.736ESM-1b
Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, not conserved), unitless0.7580.713ESM-2
Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, not conserved), unitless0.6330.512ESM-2
Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, poorly conserved), unitless0.6520.772ESM-1b
Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, poorly conserved), unitless0.550.654ESM-1b
Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs), unitless0.8270.957ESM-1b
Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs), unitless0.6770.923ESM-1b
Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, far from splice site), unitless0.8220.957ESM-1b
Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, far from splice site), unitless0.6780.926ESM-1b
Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, highly conserved), unitless0.8950.967ESM-1b
Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, highly conserved), unitless0.6570.889ESM-1b
Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, moderately conserved), unitless0.620.856ESM-1b
Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, moderately conserved), unitless0.7060.903ESM-1b
Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, near splice site), unitless0.8960.966ESM-1b
Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, near splice site), unitless0.680.895ESM-1b
Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, not conserved), unitless0.4930.802ESM-1b
Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, not conserved), unitless0.7180.899ESM-1b
Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, poorly conserved), unitless0.5030.853ESM-1b
Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, poorly conserved), unitless0.6960.937ESM-1b
Comparisons in HyCell CITATION.cff (28)
ProtocolDatasetMetricESM-2ESM-1bHigher
AAV low-vs-high (HyPLM report)FLIP AAV, low-vs-high split (FLIP split)Spearman (Spearman correlation on the FLIP AAV low-vs-high split), unitless0.2790.28Tie
AAV two-vs-many (HyPLM report)FLIP AAV, 2-vs-rest split (FLIP split)Spearman (Spearman correlation on the FLIP AAV two-vs-many split), unitless0.5180.544ESM-1b
Amylase by mutation count (HyPLM report)Amylase by mutation count (FLIP2, HyPLM report)Spearman (Spearman correlation on the FLIP2 amylase split partitioned by mutation count), unitless0.5750.503ESM-2
Amylase one-to-many (HyPLM report)Amylase one-to-many (FLIP2, HyPLM report)Spearman (Spearman correlation on the FLIP2 amylase one-to-many split), unitless0.640.653ESM-1b
Arch retrieval (DGEB) (HyPLM report)DGEB Arch retrieval (HyPLM report)Mean average precision at 5 (Zero-shot retrieval mean average precision at 5 on DGEB Arch, ranked by cosine similarity between mean-pooled representations), fraction0.120.206ESM-1b
Contacts, CASP14 (HyPLM report)CASP14 evaluation domains (82, HyPLM report)Precision at l 5 (Long-range contact precision of the top L/5 predictions (sequence separation at least 24) on 82 CASP14 evaluation domains), fraction0.3660.322ESM-2
Contacts, CASP15 (HyPLM report)CASP15 evaluation domains (95, HyPLM report)Precision at l 5 (Long-range contact precision of the top L/5 predictions (sequence separation at least 24) on 95 CASP15 evaluation domains), fraction0.3910.404ESM-1b
Contacts, ProteinNet (HyPLM report)ProteinNet CASP12Precision at l 5 (Long-range contact precision of the top L/5 predictions (sequence separation at least 24) on the ProteinNet CASP12 test set of 40 proteins), fraction0.3920.41ESM-1b
EC classification (DGEB) (HyPLM report)DGEB EC classification (HyPLM report)Macro F1 (Macro-averaged F1 of Enzyme Commission classification on DGEB, with the layer and regularisation chosen by 5-fold cross-validation on the training split because this task has no validation split), fraction0.6650.673ESM-1b
EC, DeepFRI split (HyPLM report)EC annotation, DeepFRI split (HyPLM report)F1 max (Protein-centric Fmax over Enzyme Commission numbers on the DeepFRI EC split), fraction0.8740.878Tie
EC, SaProt split (HyPLM report)EC annotation, SaProt split (HyPLM report)F1 max (Protein-centric Fmax over Enzyme Commission numbers on the SaProt EC split), fraction0.8880.889Tie
Fluorescence (HyPLM report)Fluorescence (TAPE)Spearman (Spearman correlation on the TAPE fluorescence regression split), unitless0.6640.659Tie
GB1 low-vs-high (HyPLM report)FLIP GB1, low-vs-high split (FLIP split)Spearman (Spearman correlation on the FLIP GB1 low-vs-high split), unitless0.3770.417ESM-1b
GO-BP (HyPLM report)GO biological process, DeepFRI split via SaProt (HyPLM report)F1 max (Protein-centric Fmax over Gene Ontology biological-process terms on the DeepFRI GO split as packaged by SaProt), fraction0.4960.485ESM-2
GO-CC (HyPLM report)GO cellular component, DeepFRI split via SaProt (HyPLM report)F1 max (Protein-centric Fmax over Gene Ontology cellular-component terms on the DeepFRI GO split as packaged by SaProt), fraction0.5350.52ESM-2
GO-MF (HyPLM report)GO molecular function, DeepFRI split via SaProt (HyPLM report)F1 max (Protein-centric Fmax over Gene Ontology molecular-function terms on the DeepFRI GO split as packaged by SaProt), fraction0.6670.659ESM-2
HumanPPI (HyPLM report)HumanPPI via PEER (HyPLM report)Accuracy (Human protein-protein interaction classification accuracy on PEER HumanPPI, with pair features [a, b, a*b, |a-b|]), fraction0.8560.867ESM-1b
Localization, 10-class (HyPLM report)DeepLoc ten-class localisation via PEER (HyPLM report)Accuracy (Ten-class subcellular localisation accuracy on DeepLoc 1.0 via PEER), fraction0.8060.793ESM-2
Localization, binary (HyPLM report)DeepLoc binary localisation via PEER (HyPLM report)Accuracy (Binary localisation accuracy on DeepLoc 1.0 via PEER), fraction0.9070.922ESM-1b
Meltome human cell (HyPLM report)FLIP Thermostability, Human-Cell split (FLIP split)Spearman (Spearman correlation on the FLIP Meltome human-cell split), unitless0.6960.698Tie
Meltome mixed (HyPLM report)FLIP Thermostability, Mixed split (FLIP split)Spearman (Spearman correlation on the FLIP Meltome mixed split), unitless0.6330.593ESM-2
Metal-ion binding (HyPLM report)Metal-ion binding, SaProt packaging (HyPLM report)Accuracy (Metal-ion binding classification accuracy on the split packaged by SaProt), fraction0.7250.69ESM-2
NLL, held-out UniRef panel (HyPLM report)Held-out UniRef panel (479 proteins, HyPLM report)Log loss (Mean negative log-likelihood of masked residues in nats on the report's held-out UniRef panel, identical masks and crops for every model), unitless1.821.86ESM-2
ProteinGym zero-shot (HyPLM report)ProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (Spearman correlation averaged over the 217 substitution assays of ProteinGym v1.3 with the benchmark's official aggregation, from masked-marginal zero-shot scores; per-assay values rounded to three decimals before aggregation), unitless0.4150.389ESM-2
Remote homology (fold) (HyPLM report)Fold classification (TAPE)Accuracy (Fold-level classification accuracy on the TAPE remote-homology (SCOP) test set), fraction0.3730.362ESM-2
Secondary structure (CB513) (HyPLM report)Secondary structure prediction (TAPE)Accuracy (Three-state (Q3) per-residue secondary-structure accuracy on the CB513 test set), fraction0.8440.814ESM-2
Stability (HyPLM report)Stability (TAPE, HyPLM report)Spearman (Spearman correlation on the TAPE stability regression split), unitless0.6390.596ESM-2
Thermostability (HyPLM report)Thermostability, Meltome human cell via SaProt (HyPLM report)Spearman (Spearman correlation on the Meltome human-cell split as packaged by SaProt, which the report lists separately from its own FLIP Meltome human-cell row), unitless0.6970.699Tie
Comparisons in ProteinGym (19)
ProtocolDatasetMetricESM-2ESM-1bHigher
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless0.4050.428ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless0.3170.287ESM-2
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless0.4050.406Tie
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.4670.48ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless0.4320.435Tie
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.3410.365ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.4220.396ESM-2
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless0.3870.349ESM-2
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless0.4670.491ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless0.4360.438Tie
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless0.4880.5ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless0.3330.26ESM-2
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless0.40.394ESM-2
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless0.4050.383ESM-2
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2020.226ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2190.172ESM-2
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.1730.15ESM-2
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2340.27ESM-1b
ProteinGym AMFR official zero-shot Spearman snapshotProteinGym AMFR 4G3O official snapshot cohortSpearman, unitless0.260.374ESM-1b
Comparisons in Livesey and Marsh 2025 (7)
ProtocolDatasetMetricESM-2ESM-1bHigher
Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missenseLivesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent51.454.3ESM-1b
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent72.265ESM-2
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent27.814.7ESM-2
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent74.262.4ESM-2
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent70.867ESM-2
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent70.564.5ESM-2
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filledLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent71.165.9ESM-2
Comparisons in Genomic heterogeneity inflates… (5)

What disagrees

studies
2
Compared
ESM-2, ESM-1b
Study that disagrees
Other studies
Rule
Two-study conflict; different-task
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-cbcad13670a11473 · All contested results · How findings are made and followed up