Contested result
ESM-1v and DeepSequence: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree
- Low severity
- Not yet reviewed
- two-study-conflict
- different-task
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks DeepSequence above ESM-1v in 13 of 17 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1v above DeepSequence in 6 of 7 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
2 papers rank ESM-1v above DeepSequence in 10 of their 26 comparisons and DeepSequence higher in 14; 2 are ties. Of those papers, ProteinGym ranks DeepSequence higher on its own (13 of 19).
- Rank ESM-1v higher (1): Livesey and Marsh 2025
- Rank DeepSequence higher (1): ProteinGym
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks ESM-1v higher | Ranks DeepSequence higher | Ties |
|---|---|---|---|---|
| ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and 6 more records · DOI | Agrees | 4 | 13 | 2 |
| Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and 1 more record · DOI | Agrees | 6 | 1 | 0 |
Comparisons in ProteinGym (19)
| Protocol | Dataset | Metric | ESM-1v | DeepSequence | Higher |
|---|---|---|---|---|---|
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless | 0.42 | 0.455 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless | 0.32 | 0.363 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless | 0.429 | 0.39 | ESM-1v |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.495 | 0.474 | ESM-1v |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless | 0.458 | 0.443 | ESM-1v |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.316 | 0.37 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.409 | 0.431 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless | 0.386 | 0.411 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless | 0.464 | 0.486 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless | 0.413 | 0.439 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless | 0.477 | 0.476 | Tie |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless | 0.294 | 0.358 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless | 0.407 | 0.419 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.403 | 0.405 | Tie |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.215 | 0.263 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.173 | 0.314 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.153 | 0.322 | DeepSequence |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.218 | 0.378 | DeepSequence |
| ProteinGym AMFR official zero-shot Spearman snapshot | ProteinGym AMFR 4G3O official snapshot cohort | Spearman, unitless | 0.423 | 0.407 | ESM-1v |
Comparisons in Livesey and Marsh 2025 (7)
| Protocol | Dataset | Metric | ESM-1v | DeepSequence | Higher |
|---|---|---|---|---|---|
| Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense | Livesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent | 61.4 | 65.9 | DeepSequence |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent | 81.5 | 76.6 | ESM-1v |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent | 28.1 | 17.3 | ESM-1v |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent | 78.4 | 73.5 | ESM-1v |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent | 83.6 | 78.4 | ESM-1v |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent | 81.5 | 78.4 | ESM-1v |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filled | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent | 81.8 | 77.5 | ESM-1v |
What disagrees
- studies
- 2
- Compared
- ESM-1v, DeepSequence
- Study that disagrees
- Other studies
- Rule
- two-study-conflict; different-task
- Model identity
- Matched across studies partly through links not yet reviewed
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-2eecdc1a21f0ad8f · All contested results · How findings are made and followed up