rewirebio.iobenchmarks
Contested result

ESM-1v and ESM-1b: Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) ranks ESM-1b above ESM-1v in 5 of 5 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1v above ESM-1b in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

3 papers rank ESM-1v above ESM-1b in 18 of their 31 comparisons and ESM-1b higher in 12; 1 is a tie. Of those papers, Genomic heterogeneity inflates… ranks ESM-1b higher on its own (5 of 5).

Papers grouped by which system they rank higher: ESM-1v or ESM-1bRank ESM-1v higher: Livesey and Marsh 2025. Rank ESM-1b higher: Genomic heterogeneity inflates…. Mixed: ProteinGym.Rank ESM-1v higherMixedRank ESM-1b higherLivesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and DesignGenomic heterogeneityinflates…Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)
  • Rank ESM-1v higher (1): Livesey and Marsh 2025
  • Rank ESM-1b higher (1): Genomic heterogeneity inflates…
  • Mixed (1): ProteinGym

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores ESM-1v or ESM-1b higherProteinGym: ESM-1v higher in 11, ESM-1b higher in 7, 1 tie. Livesey and Marsh 2025: ESM-1v higher in 7, ESM-1b higher in 0, 0 ties. Genomic heterogeneity inflates…: ESM-1v higher in 0, ESM-1b higher in 5, 0 ties.ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design11 ESM-1v higher111 tie7 ESM-1b higher7Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 ESM-1v higher7Genomic heterogeneity inflates…Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)5 ESM-1b higher5
ESM-1v higherTieESM-1b higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in ProteinGym (19)
ProtocolDatasetMetricESM-1vESM-1bHigher
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless0.420.428ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless0.320.287ESM-1v
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless0.4290.406ESM-1v
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.4950.48ESM-1v
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless0.4580.435ESM-1v
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.3160.365ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.4090.396ESM-1v
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless0.3860.349ESM-1v
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless0.4640.491ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless0.4130.438ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless0.4770.5ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless0.2940.26ESM-1v
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless0.4070.394ESM-1v
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless0.4030.383ESM-1v
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2150.226ESM-1b
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.1730.172Tie
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.1530.15ESM-1v
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2180.27ESM-1b
ProteinGym AMFR official zero-shot Spearman snapshotProteinGym AMFR 4G3O official snapshot cohortSpearman, unitless0.4230.374ESM-1v
Comparisons in Livesey and Marsh 2025 (7)
ProtocolDatasetMetricESM-1vESM-1bHigher
Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missenseLivesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent61.454.3ESM-1v
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent81.565ESM-1v
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent28.114.7ESM-1v
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent78.462.4ESM-1v
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent83.667ESM-1v
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent81.564.5ESM-1v
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filledLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent81.865.9ESM-1v
Comparisons in Genomic heterogeneity inflates… (5)

What disagrees

studies
2
Compared
ESM-1v, ESM-1b
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-c6ce1463e297712f · All contested results · How findings are made and followed up