Contested result

ESM-IF1 and ProteinMPNN: proteinbench primary benchmark evidence reverses the order

High severity
Not yet reviewed
Ordering reversal
Different tasks across studies

proteinbench primary benchmark evidence ranks ProteinMPNN above ESM-IF1 in 10 of 10 comparisons; other studies rank ESM-IF1 above ProteinMPNN in 24 of 24.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank ESM-IF1 above ProteinMPNN in all 24 of their comparisons. Proteinbench primary benchmark evidence ranks ProteinMPNN above ESM-IF1 in all 10 of its comparisons.

Rank ESM-IF1 higher (2): ProteinGym AMFR assay-level Spearman…, ProteinGym
Rank ProteinMPNN higher (1): proteinbench primary benchmark evidence (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

ESM-IF1 higherTieProteinMPNN higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (3)
Paper	Role	Ranks ESM-IF1 higher	Ranks ProteinMPNN higher
proteinbench primary benchmark evidence · DOI	Disagrees	0	10
ProteinGym AMFR assay-level Spearman comparison and 4 more records · Repository	Agrees	19	0
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design · Source	Agrees	5	0

Comparisons in proteinbench primary benchmark evidence (10)

Protocol	Dataset	Metric	ESM-IF1	ProteinMPNN	Higher
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-PLDDT: De novo backbones based sequence design, length 100 pLDDT	CASP, CAMEO and de novo backbones (ProteinBench split)	Plddt, unitless	88.8	94.1	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-SCTM: De novo backbones based sequence design, length 100 scTM	CASP, CAMEO and de novo backbones (ProteinBench split)	Sctm, unitless	0.81	0.962	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-PLDDT: De novo backbones based sequence design, length 200 pLDDT	CASP, CAMEO and de novo backbones (ProteinBench split)	Plddt, unitless	69.7	89.3	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-SCTM: De novo backbones based sequence design, length 200 scTM	CASP, CAMEO and de novo backbones (ProteinBench split)	Sctm, unitless	0.635	0.945	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-PLDDT: De novo backbones based sequence design, length 300 pLDDT	CASP, CAMEO and de novo backbones (ProteinBench split)	Plddt, unitless	74.4	90.3	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-SCTM: De novo backbones based sequence design, length 300 scTM	CASP, CAMEO and de novo backbones (ProteinBench split)	Sctm, unitless	0.336	0.962	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-PLDDT: De novo backbones based sequence design, length 400 pLDDT	CASP, CAMEO and de novo backbones (ProteinBench split)	Plddt, unitless	64.6	83.8	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-SCTM: De novo backbones based sequence design, length 400 scTM	CASP, CAMEO and de novo backbones (ProteinBench split)	Sctm, unitless	0.449	0.875	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-PLDDT: De novo backbones based sequence design, length 500 pLDDT	CASP, CAMEO and de novo backbones (ProteinBench split)	Plddt, unitless	59	67.1	ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-SCTM: De novo backbones based sequence design, length 500 scTM	CASP, CAMEO and de novo backbones (ProteinBench split)	Sctm, unitless	0.462	0.568	ProteinMPNN

Comparisons in ProteinGym AMFR assay-level Spearman… (19)

Protocol	Dataset	Metric	ESM-IF1	ProteinMPNN	Higher
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless	0.368	0.197	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless	0.389	0.163	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless	0.407	0.198	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.549	0.439	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless	0.417	0.284	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.292	0.187	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless	0.427	0.271	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless	0.324	0.164	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless	0.502	0.394	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless	0.498	0.348	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless	0.624	0.565	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless	0.389	0.262	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless	0.422	0.257	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless	0.44	0.292	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.35	0.26	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.301	0.174	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.294	0.183	ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutions	ProteinGym DMS substitution assays at commit 144fe22 (217 assays)	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless	0.358	0.278	ESM-IF1
ProteinGym AMFR official zero-shot Spearman snapshot	ProteinGym AMFR 4G3O official snapshot cohort	Spearman, unitless	0.433	0.353	ESM-IF1

Comparisons in ProteinGym (5)

Protocol	Dataset	Metric	ESM-IF1	ProteinMPNN	Higher
ProteinGym ZS-SUB-AUC: Zero-shot substitutions, AUC	ProteinGym substitution DMS assays (ProteinGym split)	AUROC, unitless	0.73	0.639	ESM-IF1
ProteinGym ZS-SUB-MCC: Zero-shot substitutions, MCC	ProteinGym substitution DMS assays (ProteinGym split)	MCC, unitless	0.331	0.196	ESM-IF1
ProteinGym ZS-SUB-NDCG: Zero-shot substitutions, NDCG@10%	ProteinGym substitution DMS assays (ProteinGym split)	NDCG at 10%, unitless	0.748	0.713	ESM-IF1
ProteinGym ZS-SUB-RECALL: Zero-shot substitutions, top 10% recall	ProteinGym substitution DMS assays (ProteinGym split)	Top 10% recall, fraction	0.223	0.186	ESM-IF1
ProteinGym ZS-SUB-SPEARMAN: Zero-shot substitutions, Spearman	ProteinGym substitution DMS assays (ProteinGym split)	Spearman, unitless	0.422	0.258	ESM-IF1

What disagrees

Other studies: rank ESM-IF1 above ProteinMPNN in 24 of 24 comparisons
This study: ranks ProteinMPNN above ESM-IF1 in 10 of 10 comparisons

Compared: ESM-IF1, ProteinMPNN
Study that disagrees: proteinbench primary benchmark evidence
Other studies: 6 studies
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design, ProteinGym AMFR assay-level Spearman comparison, ProteinGym clinical supervised substitutions AUC summary at commit 144fe22, ProteinGym clinical zero-shot indels AUC summary at commit 144fe22, ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22, ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies through reviewed links

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (20)
Study	Tool	Protocol or benchmark	Metric	Value
proteinbench primary benchmark evidence	ESM-IF1	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-PLDDT: De novo backbones based sequence design, length 100 pLDDT	Plddt	88.8
proteinbench primary benchmark evidence	ESM-IF1	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-SCTM: De novo backbones based sequence design, length 100 scTM	Sctm	0.81
proteinbench primary benchmark evidence	ESM-IF1	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-PLDDT: De novo backbones based sequence design, length 200 pLDDT	Plddt	69.7
proteinbench primary benchmark evidence	ESM-IF1	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-SCTM: De novo backbones based sequence design, length 200 scTM	Sctm	0.635
proteinbench primary benchmark evidence	ESM-IF1	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-PLDDT: De novo backbones based sequence design, length 300 pLDDT	Plddt	74.4
proteinbench primary benchmark evidence	ESM-IF1	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-SCTM: De novo backbones based sequence design, length 300 scTM	Sctm	0.336
proteinbench primary benchmark evidence	ESM-IF1	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-PLDDT: De novo backbones based sequence design, length 400 pLDDT	Plddt	64.6
proteinbench primary benchmark evidence	ESM-IF1	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-SCTM: De novo backbones based sequence design, length 400 scTM	Sctm	0.449
proteinbench primary benchmark evidence	ESM-IF1	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-PLDDT: De novo backbones based sequence design, length 500 pLDDT	Plddt	59
proteinbench primary benchmark evidence	ESM-IF1	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-SCTM: De novo backbones based sequence design, length 500 scTM	Sctm	0.462
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-PLDDT: De novo backbones based sequence design, length 100 pLDDT	Plddt	94.1
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-SCTM: De novo backbones based sequence design, length 100 scTM	Sctm	0.962
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-PLDDT: De novo backbones based sequence design, length 200 pLDDT	Plddt	89.3
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-SCTM: De novo backbones based sequence design, length 200 scTM	Sctm	0.945
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-PLDDT: De novo backbones based sequence design, length 300 pLDDT	Plddt	90.3
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-SCTM: De novo backbones based sequence design, length 300 scTM	Sctm	0.962
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-PLDDT: De novo backbones based sequence design, length 400 pLDDT	Plddt	83.8
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-SCTM: De novo backbones based sequence design, length 400 scTM	Sctm	0.875
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-PLDDT: De novo backbones based sequence design, length 500 pLDDT	Plddt	67.1
proteinbench primary benchmark evidence	ProteinMPNN	ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-SCTM: De novo backbones based sequence design, length 500 scTM	Sctm	0.568

In the other studies (48 results)

In the other studies
Study	Tool	Protocol or benchmark	Metric	Value
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	ESM-IF1	ProteinGym ZS-SUB-AUC: Zero-shot substitutions, AUC	AUROC	0.73
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	ESM-IF1	ProteinGym ZS-SUB-MCC: Zero-shot substitutions, MCC	MCC	0.331
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	ESM-IF1	ProteinGym ZS-SUB-NDCG: Zero-shot substitutions, NDCG@10%	NDCG at 10%	0.748
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	ESM-IF1	ProteinGym ZS-SUB-RECALL: Zero-shot substitutions, top 10% recall	Top 10% recall	0.223
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	ESM-IF1	ProteinGym ZS-SUB-SPEARMAN: Zero-shot substitutions, Spearman	Spearman	0.422
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	ProteinMPNN	ProteinGym ZS-SUB-AUC: Zero-shot substitutions, AUC	AUROC	0.639
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	ProteinMPNN	ProteinGym ZS-SUB-MCC: Zero-shot substitutions, MCC	MCC	0.196
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	ProteinMPNN	ProteinGym ZS-SUB-NDCG: Zero-shot substitutions, NDCG@10%	NDCG at 10%	0.713
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	ProteinMPNN	ProteinGym ZS-SUB-RECALL: Zero-shot substitutions, top 10% recall	Top 10% recall	0.186
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	ProteinMPNN	ProteinGym ZS-SUB-SPEARMAN: Zero-shot substitutions, Spearman	Spearman	0.258
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym AMFR official zero-shot Spearman snapshot	Spearman	0.433
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym AMFR official zero-shot Spearman snapshot	Spearman	0.353
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))	0.422
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)	0.44
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.35
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.301
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.294
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.358
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman)	0.368
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)	0.389
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)	0.407
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)	0.324
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)	0.624
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)	0.549
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)	0.292
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)	0.427
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)	0.417
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)	0.502
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.498
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ESM-IF1 [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)	0.389
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))	0.257
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)	0.292
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.26
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.174
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.183
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)	0.278
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman)	0.197
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)	0.163
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)	0.198
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)	0.164
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)	0.565
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)	0.439
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)	0.187
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)	0.271
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)	0.284
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)	0.394
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)	0.348
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py	ProteinMPNN [ProteinGym 144fe22b snapshot]	ProteinGym v1.3 zero-shot DMS substitutions	Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)	0.262

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-d734930095432234 · All contested results · How findings are made and followed up