Contested result

CARP and VESPA: ProteinGym AMFR assay-level Spearman comparison reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

ProteinGym AMFR assay-level Spearman comparison ranks VESPA above CARP in 18 of 19 comparisons; other studies rank CARP above VESPA in 6 of 6.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Livesey and Marsh 2025 ranks CARP above VESPA in all 6 of its comparisons. ProteinGym AMFR assay-level Spearman… ranks VESPA above CARP in 18 of its 19 comparisons and CARP higher in 1.

Rank CARP higher (1): Livesey and Marsh 2025
Rank VESPA higher (1): ProteinGym AMFR assay-level Spearman… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

CARP higherTieVESPA higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks CARP higher	Ranks VESPA higher	Ties
ProteinGym AMFR assay-level Spearman comparison and 4 more records · Repository	Disagrees	1	18	0
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and 1 more record · DOI	Agrees	6	0	0

Comparisons in ProteinGym AMFR assay-level Spearman… (19)

Protocol	Dataset	Metric	CARP	VESPA	Higher
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless	0.395	0.468	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless	0.282	0.366	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless	0.397	0.404	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.424	0.484	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless	0.417	0.44	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.331	0.42	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.374	0.455	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless	0.366	0.439	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless	0.395	0.503	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless	0.381	0.474	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless	0.412	0.5	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless	0.284	0.443	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless	0.369	0.435	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless	0.391	0.435	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.213	0.181	CARP
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.172	0.348	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.159	0.308	VESPA
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.162	0.328	VESPA
ProteinGym AMFR official zero-shot Spearman snapshot	ProteinGym AMFR 4G3O official snapshot cohort	Spearman, unitless	0.165	0.254	VESPA

Comparisons in Livesey and Marsh 2025 (6)

Protocol	Dataset	Metric	CARP	VESPA	Higher
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets), percent	76.1	71.3	CARP
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets), percent	20	17.3	CARP
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only), percent	73	69.5	CARP
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only), percent	77.7	72.6	CARP
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only), percent	77.2	69.5	CARP
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent	71.2	64.8	CARP

What disagrees

Other studies: rank CARP above VESPA in 6 of 6 comparisons
This study: ranks VESPA above CARP in 18 of 19 comparisons

Compared: CARP, VESPA
Study that disagrees: ProteinGym AMFR assay-level Spearman comparison, ProteinGym clinical supervised substitutions AUC summary at commit 144fe22, ProteinGym clinical zero-shot indels AUC summary at commit 144fe22, ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22, ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22
Other studies: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML), Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8)
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (38)
Study	Tool	Protocol or benchmark	Metric	Value
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym AMFR official zero-shot Spearman snapshot	Spearman	0.165
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym AMFR official zero-shot Spearman snapshot	Spearman	0.254
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))	0.369
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)	0.391
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.213
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.172
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.159
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.162
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman)	0.395
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)	0.397
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)	0.366
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)	0.412
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)	0.424
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)	0.331
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)	0.374
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)	0.417
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)	0.395
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.381
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)	0.284
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (76M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)	0.282
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))	0.435
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)	0.435
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.181
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.348
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.308
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.328
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman)	0.468
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)	0.366
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)	0.404
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)	0.439
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)	0.5
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)	0.484
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)	0.42
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)	0.455
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)	0.44
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)	0.503
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.474
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	VESPA [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)	0.443

In the other studies (12)
Study	Tool	Protocol or benchmark	Metric	Value
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets)	76.1
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets)	20
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	VESPA (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets)	71.3
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	VESPA (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets)	17.3
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only)	73
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	VESPA (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only)	69.5
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only)	77.7
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	VESPA (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only)	72.6
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only)	77.2
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score)	71.2
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	VESPA (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score)	64.8
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	VESPA (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only)	69.5

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-f34f96162a2d8bcd · All contested results · How findings are made and followed up