Contested result

ESM3 and ProteinMPNN: ProteinGym AMFR assay-level Spearman comparison reverses the order

Medium severity
Not yet reviewed
Ordering reversal
Different tasks across studies

ProteinGym AMFR assay-level Spearman comparison ranks ESM3 above ProteinMPNN in 18 of 19 comparisons; other studies rank ProteinMPNN above ESM3 in 10 of 10.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Proteinbench primary benchmark evidence ranks ProteinMPNN above ESM3 in all 10 of its comparisons. ProteinGym AMFR assay-level Spearman… ranks ESM3 above ProteinMPNN in 18 of its 19 comparisons and ProteinMPNN higher in 1.

Rank ProteinMPNN higher (1): proteinbench primary benchmark evidence
Rank ESM3 higher (1): ProteinGym AMFR assay-level Spearman… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

ProteinMPNN higherTieESM3 higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks ProteinMPNN higher	Ranks ESM3 higher	Ties
ProteinGym AMFR assay-level Spearman comparison and 4 more records · Repository	Disagrees	1	18	0
proteinbench primary benchmark evidence · DOI	Agrees	10	0	0

Comparisons in ProteinGym AMFR assay-level Spearman… (19)

Protocol	Dataset	Metric	ProteinMPNN	ESM3	Higher
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless	0.197	0.43	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless	0.163	0.4	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless	0.198	0.47	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.439	0.575	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless	0.284	0.48	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.187	0.397	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.271	0.466	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless	0.164	0.389	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless	0.394	0.548	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless	0.348	0.53	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless	0.565	0.641	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless	0.262	0.406	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless	0.257	0.466	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless	0.292	0.487	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.26	0.337	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.174	0.307	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.183	0.285	ESM3
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.278	0.363	ESM3
ProteinGym AMFR official zero-shot Spearman snapshot	ProteinGym AMFR 4G3O official snapshot cohort	Spearman, unitless	0.353	0.29	ProteinMPNN

Comparisons in proteinbench primary benchmark evidence (10)

Protocol	Dataset	Metric	ProteinMPNN	ESM3	Higher
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-PLDDT: De novo backbones based sequence design, length 100 pLDDT	CASP, CAMEO and de novo backbones (ProteinBench split)	Plddt, unitless	94.1	86.6	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-SCTM: De novo backbones based sequence design, length 100 scTM	CASP, CAMEO and de novo backbones (ProteinBench split)	Sctm, unitless	0.962	0.942	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-PLDDT: De novo backbones based sequence design, length 200 pLDDT	CASP, CAMEO and de novo backbones (ProteinBench split)	Plddt, unitless	89.3	60.7	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-SCTM: De novo backbones based sequence design, length 200 scTM	CASP, CAMEO and de novo backbones (ProteinBench split)	Sctm, unitless	0.945	0.486	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-PLDDT: De novo backbones based sequence design, length 300 pLDDT	CASP, CAMEO and de novo backbones (ProteinBench split)	Plddt, unitless	90.3	70.8	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-SCTM: De novo backbones based sequence design, length 300 scTM	CASP, CAMEO and de novo backbones (ProteinBench split)	Sctm, unitless	0.962	0.632	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-PLDDT: De novo backbones based sequence design, length 400 pLDDT	CASP, CAMEO and de novo backbones (ProteinBench split)	Plddt, unitless	83.8	62.6	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-SCTM: De novo backbones based sequence design, length 400 scTM	CASP, CAMEO and de novo backbones (ProteinBench split)	Sctm, unitless	0.875	0.564	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-PLDDT: De novo backbones based sequence design, length 500 pLDDT	CASP, CAMEO and de novo backbones (ProteinBench split)	Plddt, unitless	67.1	59.4	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-SCTM: De novo backbones based sequence design, length 500 scTM	CASP, CAMEO and de novo backbones (ProteinBench split)	Sctm, unitless	0.568	0.452	ProteinMPNN

What disagrees

Other studies: rank ProteinMPNN above ESM3 in 10 of 10 comparisons
This study: ranks ESM3 above ProteinMPNN in 18 of 19 comparisons

Compared: ESM3, ProteinMPNN
Study that disagrees: ProteinGym AMFR assay-level Spearman comparison, ProteinGym clinical supervised substitutions AUC summary at commit 144fe22, ProteinGym clinical zero-shot indels AUC summary at commit 144fe22, ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22, ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22
Other studies: proteinbench primary benchmark evidence
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies through reviewed links

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (38)
Study	Tool	Protocol or benchmark	Metric	Value
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym AMFR official zero-shot Spearman snapshot	Spearman	0.29
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym AMFR official zero-shot Spearman snapshot	Spearman	0.353
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))	0.466
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)	0.487
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.337
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.307
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.285
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.363
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman)	0.43
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)	0.4
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)	0.47
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)	0.389
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)	0.641
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)	0.575
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)	0.397
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)	0.466
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)	0.48
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)	0.548
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.53
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM3 open (1.4B) [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)	0.406
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))	0.257
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)	0.292
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.26
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.174
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.183
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.278
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman)	0.197
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)	0.163
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)	0.198
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)	0.164
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)	0.565
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)	0.439
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)	0.187
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)	0.271
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)	0.284
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)	0.394
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.348
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)	0.262

In the other studies (20)
Study	Tool	Protocol or benchmark	Metric	Value
proteinbench primary benchmark evidence	ESM3	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-PLDDT: De novo backbones based sequence design, length 100 pLDDT	Plddt	86.6
proteinbench primary benchmark evidence	ESM3	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-SCTM: De novo backbones based sequence design, length 100 scTM	Sctm	0.942
proteinbench primary benchmark evidence	ESM3	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-PLDDT: De novo backbones based sequence design, length 200 pLDDT	Plddt	60.7
proteinbench primary benchmark evidence	ESM3	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-SCTM: De novo backbones based sequence design, length 200 scTM	Sctm	0.486
proteinbench primary benchmark evidence	ESM3	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-PLDDT: De novo backbones based sequence design, length 300 pLDDT	Plddt	70.8
proteinbench primary benchmark evidence	ESM3	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-SCTM: De novo backbones based sequence design, length 300 scTM	Sctm	0.632
proteinbench primary benchmark evidence	ESM3	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-PLDDT: De novo backbones based sequence design, length 400 pLDDT	Plddt	62.6
proteinbench primary benchmark evidence	ESM3	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-SCTM: De novo backbones based sequence design, length 400 scTM	Sctm	0.564
proteinbench primary benchmark evidence	ESM3	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-PLDDT: De novo backbones based sequence design, length 500 pLDDT	Plddt	59.4
proteinbench primary benchmark evidence	ESM3	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-SCTM: De novo backbones based sequence design, length 500 scTM	Sctm	0.452
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-PLDDT: De novo backbones based sequence design, length 100 pLDDT	Plddt	94.1
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-SCTM: De novo backbones based sequence design, length 100 scTM	Sctm	0.962
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-PLDDT: De novo backbones based sequence design, length 200 pLDDT	Plddt	89.3
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-SCTM: De novo backbones based sequence design, length 200 scTM	Sctm	0.945
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-PLDDT: De novo backbones based sequence design, length 300 pLDDT	Plddt	90.3
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-SCTM: De novo backbones based sequence design, length 300 scTM	Sctm	0.962
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-PLDDT: De novo backbones based sequence design, length 400 pLDDT	Plddt	83.8
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-SCTM: De novo backbones based sequence design, length 400 scTM	Sctm	0.875
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-PLDDT: De novo backbones based sequence design, length 500 pLDDT	Plddt	67.1
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-SCTM: De novo backbones based sequence design, length 500 scTM	Sctm	0.568

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-fed245c475fd08dc · All contested results · How findings are made and followed up