Contested result

ESM-2 and ESM-1b: Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) ranks ESM-1b above ESM-2 in 26 of 28 comparisons; other studies rank ESM-2 above ESM-1b in 20 of 25.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank ESM-2 above ESM-1b in 20 of their 26 comparisons and ESM-1b higher in 5; 1 is a tie. Genome modelling and design across all… ranks ESM-1b above ESM-2 in 26 of its 28 comparisons and ESM-2 higher in 2.

Rank ESM-2 higher (2): ProteinGym AMFR assay-level Spearman…, Livesey and Marsh 2025
Rank ESM-1b higher (1): Genome modelling and design across all… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

ESM-2 higherTieESM-1b higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (3)
Paper	Role	Ranks ESM-2 higher	Ranks ESM-1b higher	Ties
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) and 1 more record · DOI	Disagrees	2	26	0
ProteinGym AMFR assay-level Spearman comparison and 4 more records · Repository	Agrees	14	4	1
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and 1 more record · DOI	Agrees	6	1	0

Comparisons in Genome modelling and design across all… (28)

Protocol	Dataset	Metric	ESM-2	ESM-1b	Higher
Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding indels), unitless	0.772	0.931	ESM-1b
Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding indels), unitless	0.533	0.809	ESM-1b
Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding microsatellite variants), unitless	0.253	0.692	ESM-1b
Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding microsatellite variants), unitless	0.542	0.814	ESM-1b
Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding non-SNVs, far from splice site), unitless	0.741	0.92	ESM-1b
Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding non-SNVs, far from splice site), unitless	0.531	0.808	ESM-1b
Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding non-SNVs, highly conserved), unitless	0.946	0.986	ESM-1b
Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding non-SNVs, highly conserved), unitless	0.51	0.832	ESM-1b
Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding non-SNVs, moderately conserved), unitless	0.735	0.897	ESM-1b
Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding non-SNVs, moderately conserved), unitless	0.502	0.736	ESM-1b
Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding non-SNVs, not conserved), unitless	0.758	0.713	ESM-2
Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding non-SNVs, not conserved), unitless	0.633	0.512	ESM-2
Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding non-SNVs, poorly conserved), unitless	0.652	0.772	ESM-1b
Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding non-SNVs, poorly conserved), unitless	0.55	0.654	ESM-1b
Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding SNVs), unitless	0.827	0.957	ESM-1b
Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding SNVs), unitless	0.677	0.923	ESM-1b
Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding SNVs, far from splice site), unitless	0.822	0.957	ESM-1b
Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding SNVs, far from splice site), unitless	0.678	0.926	ESM-1b
Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding SNVs, highly conserved), unitless	0.895	0.967	ESM-1b
Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding SNVs, highly conserved), unitless	0.657	0.889	ESM-1b
Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding SNVs, moderately conserved), unitless	0.62	0.856	ESM-1b
Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding SNVs, moderately conserved), unitless	0.706	0.903	ESM-1b
Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding SNVs, near splice site), unitless	0.896	0.966	ESM-1b
Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding SNVs, near splice site), unitless	0.68	0.895	ESM-1b
Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding SNVs, not conserved), unitless	0.493	0.802	ESM-1b
Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding SNVs, not conserved), unitless	0.718	0.899	ESM-1b
Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUPRC (zero-shot; ClinVar coding SNVs, poorly conserved), unitless	0.503	0.853	ESM-1b
Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	ClinVar October 2024 release, filtered (Brixi et al. 2026)	AUROC (zero-shot; ClinVar coding SNVs, poorly conserved), unitless	0.696	0.937	ESM-1b

Comparisons in ProteinGym AMFR assay-level Spearman… (19)

Protocol	Dataset	Metric	ESM-2	ESM-1b	Higher
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless	0.425	0.428	Tie
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless	0.337	0.287	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless	0.415	0.406	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.513	0.48	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless	0.457	0.435	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.341	0.365	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.423	0.396	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless	0.387	0.349	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless	0.486	0.491	ESM-1b
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless	0.458	0.438	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless	0.523	0.5	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless	0.333	0.26	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless	0.414	0.394	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless	0.422	0.383	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.248	0.226	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.219	0.172	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.173	0.15	ESM-2
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.234	0.27	ESM-1b
ProteinGym AMFR official zero-shot Spearman snapshot	ProteinGym AMFR 4G3O official snapshot cohort	Spearman, unitless	0.321	0.374	ESM-1b

Comparisons in Livesey and Marsh 2025 (7)

Protocol	Dataset	Metric	ESM-2	ESM-1b	Higher
Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense	Livesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent	51.4	54.3	ESM-1b
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets), percent	72.2	65	ESM-2
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets), percent	27.8	14.7	ESM-2
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only), percent	74.2	62.4	ESM-2
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only), percent	70.8	67	ESM-2
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only), percent	70.5	64.5	ESM-2
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent	71.1	65.9	ESM-2

What disagrees

Other studies: rank ESM-2 above ESM-1b in 20 of 25 comparisons
This study: ranks ESM-1b above ESM-2 in 26 of 28 comparisons

Compared: ESM-2, ESM-1b
Study that disagrees: Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026), Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026)
Other studies: 7 studies
ProteinGym AMFR assay-level Spearman comparison, Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML), Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8), ProteinGym clinical supervised substitutions AUC summary at commit 144fe22, ProteinGym clinical zero-shot indels AUC summary at commit 144fe22, ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22, ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (56)
Study	Tool	Protocol or benchmark	Metric	Value
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding indels)	0.931
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding indels)	0.809
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding microsatellite variants)	0.692
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding microsatellite variants)	0.814
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding non-SNVs, far from splice site)	0.92
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding non-SNVs, far from splice site)	0.808
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding non-SNVs, highly conserved)	0.986
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding non-SNVs, highly conserved)	0.832
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding non-SNVs, moderately conserved)	0.897
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding non-SNVs, moderately conserved)	0.736
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding non-SNVs, not conserved)	0.713
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding non-SNVs, not conserved)	0.512
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding non-SNVs, poorly conserved)	0.772
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding non-SNVs, poorly conserved)	0.654
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs)	0.957
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs)	0.923
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, far from splice site)	0.957
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, far from splice site)	0.926
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, highly conserved)	0.967
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, highly conserved)	0.889
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, moderately conserved)	0.856
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, moderately conserved)	0.903
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, near splice site)	0.966
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, near splice site)	0.895
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, not conserved)	0.802
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, not conserved)	0.899
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, poorly conserved)	0.853
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-1b (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, poorly conserved)	0.937
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding indels)	0.772
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding indels)	0.533
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding microsatellite variants)	0.253
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding microsatellite variants)	0.542
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding non-SNVs, far from splice site)	0.741
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding non-SNVs, far from splice site)	0.531
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding non-SNVs, highly conserved)	0.946
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding non-SNVs, highly conserved)	0.51
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding non-SNVs, moderately conserved)	0.735
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding non-SNVs, moderately conserved)	0.502
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding non-SNVs, not conserved)	0.758
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding non-SNVs, not conserved)	0.633
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding non-SNVs, poorly conserved)	0.652
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding non-SNVs, poorly conserved)	0.55
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs)	0.827
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs)	0.677
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, far from splice site)	0.822
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, far from splice site)	0.678
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, highly conserved)	0.895
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, highly conserved)	0.657
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, moderately conserved)	0.62
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, moderately conserved)	0.706
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, near splice site)	0.896
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, near splice site)	0.68
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, not conserved)	0.493
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, not conserved)	0.718
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	AUPRC (zero-shot; ClinVar coding SNVs, poorly conserved)	0.503
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XML	ESM-2 650M (Evo 2 paper variant effect evaluations)	Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)	AUROC (zero-shot; ClinVar coding SNVs, poorly conserved)	0.696

In the other studies (51 results)

In the other studies
Study	Tool	Protocol or benchmark	Metric	Value
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym AMFR official zero-shot Spearman snapshot	Spearman	0.374
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv	ESM2 (150M) [ProteinGym 144fe22b snapshot]	ProteinGym AMFR official zero-shot Spearman snapshot	Spearman	0.321
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense	Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each)	54.3
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM2 (Livesey and Marsh 2025)	Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense	Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each)	51.4
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets)	65
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets)	14.7
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM2 (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets)	72.2
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM2 (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets)	27.8
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only)	62.4
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM2 (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only)	74.2
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only)	67
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM2 (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only)	70.8
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only)	64.5
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM2 (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only)	70.5
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM-1b (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score)	65.9
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)	ESM2 (Livesey and Marsh 2025)	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score)	71.1
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))	0.394
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)	0.383
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.226
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.172
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.15
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.27
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)	0.287
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)	0.406
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)	0.349
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)	0.5
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)	0.48
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)	0.365
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)	0.396
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)	0.435
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)	0.491
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.438
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-1b [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)	0.26
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (15B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.219
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (15B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.173
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (15B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.234
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (15B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)	0.387
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (15B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)	0.341
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (15B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)	0.333
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (3B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)	0.423
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (3B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.458
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (650M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))	0.414
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (650M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)	0.422
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (650M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.248
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (650M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)	0.337
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (650M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)	0.415
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (650M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)	0.523
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (650M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)	0.513
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (650M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)	0.457
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (650M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)	0.486
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM2 (650M) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.458

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-3cdcde4b84b1acf3 · All contested results · How findings are made and followed up