Contested result

CARP and ESM-1b: ProteinGym AMFR assay-level Spearman comparison reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

ProteinGym AMFR assay-level Spearman comparison ranks ESM-1b above CARP in 14 of 18 comparisons; other studies rank CARP above ESM-1b in 6 of 6.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Livesey and Marsh 2025 ranks CARP above ESM-1b in all 6 of its comparisons. ProteinGym AMFR assay-level Spearman… ranks ESM-1b above CARP in 14 of its 19 comparisons and CARP higher in 4; 1 is a tie.

Rank CARP higher (1): Livesey and Marsh 2025
Rank ESM-1b higher (1): ProteinGym AMFR assay-level Spearman… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

CARP higherTieESM-1b higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks CARP higher	Ranks ESM-1b higher	Ties
ProteinGym AMFR assay-level Spearman comparison and 4 more records · Repository	Disagrees	4	14	1
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and 1 more record · DOI	Agrees	6	0	0

Comparisons in ProteinGym AMFR assay-level Spearman… (19)

Protocol	Dataset	Metric	CARP	ESM-1b	Higher
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless	0.395	0.428	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless	0.282	0.287	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless	0.397	0.406	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.424	0.48	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless	0.417	0.435	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.331	0.365	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.374	0.396	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless	0.366	0.349	CARP
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless	0.395	0.491	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless	0.381	0.438	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless	0.412	0.5	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless	0.284	0.26	CARP
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless	0.369	0.394	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless	0.391	0.383	CARP
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.213	0.226	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.172	0.172	Tie
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.159	0.15	CARP
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.162	0.27	ESM-1b
ProteinGym AMFR official zero-shot Spearman snapshot	ProteinGym AMFR 4G3O official snapshot cohort	Spearman, unitless	0.165	0.374	ESM-1b

Comparisons in Livesey and Marsh 2025 (6)

Protocol	Dataset	Metric	CARP	ESM-1b	Higher
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets), percent	76.1	65	CARP
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets), percent	20	14.7	CARP
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only), percent	73	62.4	CARP
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only), percent	77.7	67	CARP
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only), percent	77.2	64.5	CARP
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent	71.2	65.9	CARP

What disagrees

Other studies: rank CARP above ESM-1b in 6 of 6 comparisons
This study: ranks ESM-1b above CARP in 14 of 18 comparisons

Compared: CARP, ESM-1b
Study that disagrees: ProteinGym AMFR assay-level Spearman comparison, ProteinGym clinical supervised substitutions AUC summary at commit 144fe22, ProteinGym clinical zero-shot indels AUC summary at commit 144fe22, ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22, ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22
Other studies: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML), Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8)
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (36)
Study	Tool	Protocol or benchmark	Metric	Value
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym AMFR official zero-shot Spearman snapshot	Spearman	0.165
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym AMFR official zero-shot Spearman snapshot	Spearman	0.374
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))	0.369
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)	0.391
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.213
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.159
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.162
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman)	0.395
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)	0.397
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)	0.366
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)	0.412
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)	0.424
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)	0.331
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)	0.374
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)	0.417
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)	0.395
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.381
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (640M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)	0.284
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	CARP (76M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)	0.282
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))	0.394
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)	0.383
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.226
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.15
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.27
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman)	0.428
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)	0.287
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)	0.406
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)	0.349
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)	0.5
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)	0.48
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)	0.365
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)	0.396
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)	0.435
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)	0.491
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.438
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)	0.26

In the other studies (12)
Study	Tool	Protocol or benchmark	Metric	Value
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets)	76.1
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets)	20
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets)	65
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets)	14.7
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only)	73
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only)	62.4
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only)	77.7
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only)	67
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only)	77.2
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only)	64.5
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	CARP (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score)	71.2
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score)	65.9

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-cd1646842db5f630 · All contested results · How findings are made and followed up