Contested result
EVmutation and CARP: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree
- Low severity
- Not yet reviewed
- two-study-conflict
- different-task
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks EVmutation above CARP in 15 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above EVmutation in 5 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
2 papers rank EVmutation above CARP in 16 of their 25 comparisons and CARP higher in 9. Of those papers, Livesey and Marsh 2025 ranks CARP higher on its own (5 of 6).
- Rank EVmutation higher (1): ProteinGym
- Rank CARP higher (1): Livesey and Marsh 2025
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks EVmutation higher | Ranks CARP higher | Ties |
|---|---|---|---|---|
| ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and 6 more records · DOI | Agrees | 15 | 4 | 0 |
| Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and 1 more record · DOI | Agrees | 1 | 5 | 0 |
Comparisons in ProteinGym (19)
| Protocol | Dataset | Metric | EVmutation | CARP | Higher |
|---|---|---|---|---|---|
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless | 0.44 | 0.395 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless | 0.317 | 0.273 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless | 0.378 | 0.397 | CARP |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.404 | 0.424 | CARP |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless | 0.41 | 0.417 | CARP |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.399 | 0.331 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.426 | 0.374 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless | 0.41 | 0.366 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless | 0.448 | 0.395 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless | 0.414 | 0.381 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless | 0.43 | 0.412 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless | 0.393 | 0.284 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless | 0.395 | 0.369 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.376 | 0.391 | CARP |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.273 | 0.213 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.327 | 0.172 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.318 | 0.159 | EVmutation |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.394 | 0.162 | EVmutation |
| ProteinGym AMFR official zero-shot Spearman snapshot | ProteinGym AMFR 4G3O official snapshot cohort | Spearman, unitless | 0.418 | 0.165 | EVmutation |
Comparisons in Livesey and Marsh 2025 (6)
| Protocol | Dataset | Metric | EVmutation | CARP | Higher |
|---|---|---|---|---|---|
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent | 70.8 | 76.1 | CARP |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent | 10.7 | 20 | CARP |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent | 74.3 | 73 | EVmutation |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent | 68.3 | 77.7 | CARP |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent | 72 | 77.2 | CARP |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filled | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent | 68.8 | 71.2 | CARP |
What disagrees
- studies
- 2
- Compared
- EVmutation, CARP
- Study that disagrees
- Other studies
- Rule
- two-study-conflict; different-task
- Model identity
- Matched across studies partly through links not yet reviewed
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-d29ee2a1bcd98988 · All contested results · How findings are made and followed up