rewirebio.iobenchmarks
Contested result

ESM-2 and ESM3: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and pfmbench primary benchmark evidence disagree

  • Medium severity
  • Under investigation
  • two-study-conflict
  • Same task, different protocols

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM3 above ESM-2 in 18 of 18 comparisons; pfmbench primary benchmark evidence ranks ESM-2 above ESM3 in 9 of 12 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank ESM-2 above ESM3 in 9 of their 31 comparisons and ESM3 higher in 21; 1 is a tie. Of those papers, ProteinGym ranks ESM3 higher on its own (18 of 19).

Papers grouped by which system they rank higher: ESM-2 or ESM3Rank ESM-2 higher: pfmbench primary benchmark evidence. Rank ESM3 higher: ProteinGym. Mixed: none.Rank ESM-2 higherMixedRank ESM3 higherpfmbench primarybenchmark evidencepfmbench primary benchmark evidenceProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design
  • Rank ESM-2 higher (1): pfmbench primary benchmark evidence
  • Rank ESM3 higher (1): ProteinGym
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores ESM-2 or ESM3 higherProteinGym: ESM-2 higher in 0, ESM3 higher in 18, 1 tie. pfmbench primary benchmark evidence: ESM-2 higher in 9, ESM3 higher in 3, 0 ties.ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design1 tie18 ESM3 higher18pfmbench primary benchmark evidencepfmbench primary benchmark evidence9 ESM-2 higher93 ESM3 higher3
ESM-2 higherTieESM3 higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks ESM-2 higherRanks ESM3 higherTies
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and 6 more records · DOIAgrees0181
pfmbench primary benchmark evidence · DOIAgrees930
Comparisons in ProteinGym (19)
ProtocolDatasetMetricESM-2ESM3Higher
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless0.4050.43ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless0.3170.4ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless0.4050.47ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.4670.575ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless0.4320.48ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.3410.397ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.4220.466ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless0.3870.389Tie
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless0.4670.548ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless0.4360.53ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless0.4880.641ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless0.3330.406ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless0.40.466ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless0.4050.487ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2020.337ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2190.307ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.1730.285ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2340.363ESM3
ProteinGym AMFR official zero-shot Spearman snapshotProteinGym AMFR 4G3O official snapshot cohortSpearman, unitless0.260.29ESM3
Comparisons in pfmbench primary benchmark evidence (12)
ProtocolDatasetMetricESM-2ESM3Higher
PFMBench ANTI-RES: Antibiotic resistanceAntibiotic resistance (PFMBench split)Accuracy, fraction0.6340.584ESM-2
PFMBench BINDING-DB: BindingDBBindingDB (PFMBench split)Spearman, unitless0.1370.225ESM3
PFMBench CLONING-CLF: Cloning CLFCloning CLF (PFMBench split)AUROC, unitless0.8060.774ESM-2
PFMBench DEEPLOC2: DeepLoc2 MultiDeepLoc2 Multi (PFMBench split)F1 max (multi-label; TorchDrug-style F1_score_max), fraction0.7620.659ESM-2
PFMBench DEEPSOL: DeepSolDeepSol (PFMBench split)AUROC, unitless0.8450.781ESM-2
PFMBench EC: Enzyme CommissionEnzyme Commission (PFMBench split)F1 max (multi-label; TorchDrug-style F1_score_max), fraction0.7360.648ESM-2
PFMBench MAT-PROD: Material productionMaterial production (PFMBench split)Accuracy, fraction0.8120.775ESM-2
PFMBench METAL-ION: Metal ion bindingMetal ion binding (PFMBench split)Accuracy, fraction0.7120.703ESM-2
PFMBench PDB-BIND: PDBbindPDBbind (PFMBench split)Spearman, unitless0.1470.156ESM3
PFMBench PROTEINGYM-ZS: ProteinGym zero-shot variant effect predictionProteinGym (PFMBench split)Spearman, unitless0.4390.414ESM-2
PFMBench SEC-STRUCT: Secondary structureSecondary structure (PFMBench split)Accuracy, fraction0.7640.813ESM3
PFMBench STABILITY: TAPE_StabilityTAPE_Stability (PFMBench split)Spearman, unitless0.3210.157ESM-2

What disagrees

studies
2
Compared
ESM-2, ESM3
Study that disagrees
Other studies
Rule
two-study-conflict; same task, different protocols
Model identity
Matched across studies through reviewed links
Earlier IDs
discrepancy-4db3e5d24233e19b

Follow-up

Current state: Under investigation. Next check 6 November 2026.

  1. 23 October 2026: Investigated · rewire-bio/rewire-benchmark-data#210

    State after this step: Under investigation

    Eleven of PFMBench's twelve comparisons are supervised probing tasks with no counterpart elsewhere. The twelfth, PFMBench's ProteinGym zero-shot row, gives ESM3 0.414 and ESM-2 0.439, while ProteinGym's own files give ESM3 open 1.4B 0.466 and ESM-2 650M 0.414. Being checked: PFMBench's assay subset, scoring rule and checkpoints (issue 210).

    Next check: 6 November 2026

  2. 24 October 2026: Under investigation · rewire-bio/rewire-benchmark-data#210

    State after this step: Under investigation

    Eleven of PFMBench's twelve comparisons are supervised probing tasks with no counterpart elsewhere. The twelfth, PFMBench's ProteinGym zero-shot row, gives ESM3 0.414 and ESM-2 0.439, while ProteinGym's own files give ESM3 open 1.4B 0.466 and ESM-2 650M 0.414. Being checked: PFMBench's assay subset, scoring rule and checkpoints (issue 210).

    Next check: 6 November 2026

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-a2ed36d1ca54b5b9 · All contested results · How findings are made and followed up