rewirebio.iobenchmarks
Contested result

ESM-IF1 and ProteinMPNN: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and proteinbench primary benchmark evidence disagree

  • Low severity
  • Explained
  • two-study-conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM-IF1 above ProteinMPNN in 19 of 19 comparisons; proteinbench primary benchmark evidence ranks ProteinMPNN above ESM-IF1 in 10 of 10 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank ESM-IF1 above ProteinMPNN in 19 of their 29 comparisons and ProteinMPNN higher in 10. Of those papers, proteinbench primary benchmark evidence ranks ProteinMPNN higher on its own (10 of 10).

Papers grouped by which system they rank higher: ESM-IF1 or ProteinMPNNRank ESM-IF1 higher: ProteinGym. Rank ProteinMPNN higher: proteinbench primary benchmark evidence. Mixed: none.Rank ESM-IF1 higherMixedRank ProteinMPNN higherProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Designproteinbench primarybenchmark evidenceproteinbench primary benchmark evidence
  • Rank ESM-IF1 higher (1): ProteinGym
  • Rank ProteinMPNN higher (1): proteinbench primary benchmark evidence
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores ESM-IF1 or ProteinMPNN higherProteinGym: ESM-IF1 higher in 19, ProteinMPNN higher in 0, 0 ties. proteinbench primary benchmark evidence: ESM-IF1 higher in 0, ProteinMPNN higher in 10, 0 ties.ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design19 ESM-IF1 higher19proteinbench primary benchmark evidenceproteinbench primary benchmark evidence10 ProteinMPNN higher10
ESM-IF1 higherTieProteinMPNN higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks ESM-IF1 higherRanks ProteinMPNN higherTies
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and 6 more records · DOIAgrees1900
proteinbench primary benchmark evidence · DOIAgrees0100
Comparisons in ProteinGym (19)
ProtocolDatasetMetricESM-IF1ProteinMPNNHigher
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless0.3680.197ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless0.3890.163ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless0.4070.198ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.5490.439ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless0.4170.284ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.2920.187ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.4270.271ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless0.3240.164ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless0.5020.394ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless0.4980.348ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless0.6240.565ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless0.3890.262ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless0.4220.257ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless0.440.292ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.350.26ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.3010.174ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2940.183ESM-IF1
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.3580.278ESM-IF1
ProteinGym AMFR official zero-shot Spearman snapshotProteinGym AMFR 4G3O official snapshot cohortSpearman, unitless0.4330.353ESM-IF1
Comparisons in proteinbench primary benchmark evidence (10)
ProtocolDatasetMetricESM-IF1ProteinMPNNHigher
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-PLDDT: De novo backbones based sequence design, length 100 pLDDTCASP, CAMEO and de novo backbones (ProteinBench split)Plddt, unitless88.894.1ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-100-SCTM: De novo backbones based sequence design, length 100 scTMCASP, CAMEO and de novo backbones (ProteinBench split)Sctm, unitless0.810.962ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-PLDDT: De novo backbones based sequence design, length 200 pLDDTCASP, CAMEO and de novo backbones (ProteinBench split)Plddt, unitless69.789.3ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-200-SCTM: De novo backbones based sequence design, length 200 scTMCASP, CAMEO and de novo backbones (ProteinBench split)Sctm, unitless0.6350.945ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-PLDDT: De novo backbones based sequence design, length 300 pLDDTCASP, CAMEO and de novo backbones (ProteinBench split)Plddt, unitless74.490.3ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-300-SCTM: De novo backbones based sequence design, length 300 scTMCASP, CAMEO and de novo backbones (ProteinBench split)Sctm, unitless0.3360.962ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-PLDDT: De novo backbones based sequence design, length 400 pLDDTCASP, CAMEO and de novo backbones (ProteinBench split)Plddt, unitless64.683.8ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-400-SCTM: De novo backbones based sequence design, length 400 scTMCASP, CAMEO and de novo backbones (ProteinBench split)Sctm, unitless0.4490.875ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-PLDDT: De novo backbones based sequence design, length 500 pLDDTCASP, CAMEO and de novo backbones (ProteinBench split)Plddt, unitless5967.1ProteinMPNN
ProteinBench IF-DE-NOVO-BACKBONES-BASED-SEQUENCE-DESIGN-LENGTH-500-SCTM: De novo backbones based sequence design, length 500 scTMCASP, CAMEO and de novo backbones (ProteinBench split)Sctm, unitless0.4620.568ProteinMPNN

What disagrees

studies
2
Compared
ESM-IF1, ProteinMPNN
Study that disagrees
Other studies
Rule
two-study-conflict; different-task
Model identity
Matched across studies through reviewed links
Earlier IDs
discrepancy-d734930095432234

Follow-up

Current state: Explained. Settled.

  1. 23 October 2026: Investigated · rewire-bio/rewire-benchmark-data#201

    State after this step: Explained

    ProteinBench measures sequence design on de novo backbones of length 100 to 500, scored by pLDDT and scTM after refolding, the task ProteinMPNN was built for. Every consensus comparison is ProteinGym zero-shot variant effect prediction (DMS Spearman, clinical AUC), where ESM-IF1 scores 0.422 against ProteinMPNN's 0.257. Design quality and variant scoring are different endpoints; both sets of results are kept as reported. The high severity comes from the strong-consensus step cancelling the cross-task step (issue 201).

  2. 24 October 2026: Explained · rewire-bio/rewire-benchmark-data#201

    State after this step: Explained

    ProteinBench measures sequence design on de novo backbones of length 100 to 500, scored by pLDDT and scTM after refolding, the task ProteinMPNN was built for. Every consensus comparison is ProteinGym zero-shot variant effect prediction (DMS Spearman, clinical AUC), where ESM-IF1 scores 0.422 against ProteinMPNN's 0.257. Design quality and variant scoring are different endpoints; both sets of results are kept as reported. The high severity comes from the strong-consensus step cancelling the cross-task step (issue 201).

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-1f1fd7fb7ee75052 · All contested results · How findings are made and followed up