rewirebio.iobenchmarks
Contested result

ESM3 and ProteinMPNN: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and proteinbench primary benchmark evidence disagree

  • Low severity
  • Explained
  • two-study-conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM3 above ProteinMPNN in 18 of 19 comparisons; proteinbench primary benchmark evidence ranks ProteinMPNN above ESM3 in 10 of 10 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank ESM3 above ProteinMPNN in 18 of their 29 comparisons and ProteinMPNN higher in 11. Of those papers, proteinbench primary benchmark evidence ranks ProteinMPNN higher on its own (10 of 10).

Papers grouped by which system they rank higher: ESM3 or ProteinMPNNRank ESM3 higher: ProteinGym. Rank ProteinMPNN higher: proteinbench primary benchmark evidence. Mixed: none.Rank ESM3 higherMixedRank ProteinMPNN higherProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Designproteinbench primarybenchmark evidenceproteinbench primary benchmark evidence
  • Rank ESM3 higher (1): ProteinGym
  • Rank ProteinMPNN higher (1): proteinbench primary benchmark evidence
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores ESM3 or ProteinMPNN higherProteinGym: ESM3 higher in 18, ProteinMPNN higher in 1, 0 ties. proteinbench primary benchmark evidence: ESM3 higher in 0, ProteinMPNN higher in 10, 0 ties.ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design18 ESM3 higher181 ProteinMPNN higherproteinbench primary benchmark evidenceproteinbench primary benchmark evidence10 ProteinMPNN higher10
ESM3 higherTieProteinMPNN higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks ESM3 higherRanks ProteinMPNN higherTies
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and 6 more records · DOIAgrees1810
proteinbench primary benchmark evidence · DOIAgrees0100
Comparisons in ProteinGym (19)
ProtocolDatasetMetricESM3ProteinMPNNHigher
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless0.430.197ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless0.40.163ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless0.470.198ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.5750.439ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless0.480.284ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.3970.187ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.4660.271ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless0.3890.164ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless0.5480.394ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless0.530.348ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless0.6410.565ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless0.4060.262ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless0.4660.257ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless0.4870.292ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.3370.26ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.3070.174ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2850.183ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.3630.278ESM3
ProteinGym AMFR official zero-shot Spearman snapshotProteinGym AMFR 4G3O official snapshot cohortSpearman, unitless0.290.353ProteinMPNN
Comparisons in proteinbench primary benchmark evidence (10)
ProtocolDatasetMetricESM3ProteinMPNNHigher
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-PLDDT: De novo backbones based sequence design, length 100 pLDDTCASP, CAMEO and de novo backbones (ProteinBench split)Plddt, unitless86.694.1ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-SCTM: De novo backbones based sequence design, length 100 scTMCASP, CAMEO and de novo backbones (ProteinBench split)Sctm, unitless0.9420.962ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-PLDDT: De novo backbones based sequence design, length 200 pLDDTCASP, CAMEO and de novo backbones (ProteinBench split)Plddt, unitless60.789.3ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-SCTM: De novo backbones based sequence design, length 200 scTMCASP, CAMEO and de novo backbones (ProteinBench split)Sctm, unitless0.4860.945ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-PLDDT: De novo backbones based sequence design, length 300 pLDDTCASP, CAMEO and de novo backbones (ProteinBench split)Plddt, unitless70.890.3ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-SCTM: De novo backbones based sequence design, length 300 scTMCASP, CAMEO and de novo backbones (ProteinBench split)Sctm, unitless0.6320.962ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-PLDDT: De novo backbones based sequence design, length 400 pLDDTCASP, CAMEO and de novo backbones (ProteinBench split)Plddt, unitless62.683.8ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-SCTM: De novo backbones based sequence design, length 400 scTMCASP, CAMEO and de novo backbones (ProteinBench split)Sctm, unitless0.5640.875ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-PLDDT: De novo backbones based sequence design, length 500 pLDDTCASP, CAMEO and de novo backbones (ProteinBench split)Plddt, unitless59.467.1ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-SCTM: De novo backbones based sequence design, length 500 scTMCASP, CAMEO and de novo backbones (ProteinBench split)Sctm, unitless0.4520.568ProteinMPNN

What disagrees

studies
2
Compared
ESM3, ProteinMPNN
Study that disagrees
Other studies
Rule
two-study-conflict; different-task
Model identity
Matched across studies through reviewed links
Earlier IDs
discrepancy-f82ed80f71c6f0bd

Follow-up

Current state: Explained. Settled.

  1. 23 October 2026: Investigated · rewire-bio/rewire-benchmark-data#205

    State after this step: Explained

    ProteinBench ranks ProteinMPNN above ESM3 on de novo backbone sequence design (pLDDT and scTM after refolding); the consensus is ProteinGym zero-shot variant effect prediction, where ESM3 leads. Different endpoints; both kept as reported. The same pair also fires the other way round as discrepancy-fed245c475fd08dc, with ProteinGym as the outlier (issue 205).

  2. 24 October 2026: Explained · rewire-bio/rewire-benchmark-data#205

    State after this step: Explained

    ProteinBench ranks ProteinMPNN above ESM3 on de novo backbone sequence design (pLDDT and scTM after refolding); the consensus is ProteinGym zero-shot variant effect prediction, where ESM3 leads. Different endpoints; both kept as reported. The same pair also fires the other way round as discrepancy-fed245c475fd08dc, with ProteinGym as the outlier (issue 205).

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-d586dd34b17efa52 · All contested results · How findings are made and followed up