rewirebio.iobenchmarks
Contested result

ESM3 relative to ESM-2: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design is outside the range of other studies

  • Low severity
  • Not yet reviewed
  • Gap outlier
  • Same task, different protocols

On ProteinGym mutation effects, ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design reports ESM3 at a median relative difference of +0.20 from ESM-2; other studies report -0.06 to -0.01.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

In 2 papers, ESM3 scores between 1% and 6% below ESM-2. In ProteinGym, ESM3 scores 20% above ESM-2.

How far ESM3 scores from ESM-2 in each paperThe other papers range from −6% to −1%. ProteinGym (disagrees): +20%. HyCell CITATION.cff: −1%. pfmbench primary benchmark evidence: −6%.ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and DesignProteinGym: +20%+20%HyCell CITATION.cffHyCell CITATION.cff (commit 0dcc533c)HyCell CITATION.cff: −1%−1%pfmbench primary benchmark evidencepfmbench primary benchmark evidencepfmbench primary benchmark evidence: −6%−6%−6%= ESM-2+20%
Range of the other papersRelative difference of ESM3 from ESM-2, (system − reference) / reference; the outlined dot is the paper that disagrees.
Papers compared (3)
PaperRoleRanks ESM3 higherRanks ESM-2 higherTiesGap to ESM-2
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and 6 more records · DOIDisagrees1801+20%
HyCell CITATION.cff (commit 0dcc533c) and 17 more records · RepositoryAgrees461−1%
pfmbench primary benchmark evidence · DOIAgrees010−6%
Comparisons in ProteinGym (19)
ProtocolDatasetMetricESM3ESM-2Higher
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless0.430.405ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless0.40.317ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless0.470.405ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.5750.467ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless0.480.432ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.3970.341ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless0.4660.422ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless0.3890.387Tie
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless0.5480.467ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless0.530.436ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless0.6410.488ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless0.4060.333ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless0.4660.4ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless0.4870.405ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.3370.202ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.3070.219ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.2850.173ESM3
ProteinGym v1.3 zero-shot DMS substitutionsProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless0.3630.234ESM3
ProteinGym AMFR official zero-shot Spearman snapshotProteinGym AMFR 4G3O official snapshot cohortSpearman, unitless0.290.26ESM3
Comparisons in HyCell CITATION.cff (11)
ProtocolDatasetMetricESM3ESM-2Higher
AAV low-vs-high (HyPLM report)FLIP AAV, low-vs-high split (FLIP split)Spearman (Spearman correlation on the FLIP AAV low-vs-high split), unitless0.2870.279ESM3
AAV two-vs-many (HyPLM report)FLIP AAV, 2-vs-rest split (FLIP split)Spearman (Spearman correlation on the FLIP AAV two-vs-many split), unitless0.5510.518ESM3
Amylase by mutation count (HyPLM report)Amylase by mutation count (FLIP2, HyPLM report)Spearman (Spearman correlation on the FLIP2 amylase split partitioned by mutation count), unitless0.5240.575ESM-2
Amylase one-to-many (HyPLM report)Amylase one-to-many (FLIP2, HyPLM report)Spearman (Spearman correlation on the FLIP2 amylase one-to-many split), unitless0.6180.64ESM-2
Fluorescence (HyPLM report)Fluorescence (TAPE)Spearman (Spearman correlation on the TAPE fluorescence regression split), unitless0.6690.664Tie
GB1 low-vs-high (HyPLM report)FLIP GB1, low-vs-high split (FLIP split)Spearman (Spearman correlation on the FLIP GB1 low-vs-high split), unitless0.4550.377ESM3
Meltome human cell (HyPLM report)FLIP Thermostability, Human-Cell split (FLIP split)Spearman (Spearman correlation on the FLIP Meltome human-cell split), unitless0.6820.696ESM-2
Meltome mixed (HyPLM report)FLIP Thermostability, Mixed split (FLIP split)Spearman (Spearman correlation on the FLIP Meltome mixed split), unitless0.5730.633ESM-2
ProteinGym zero-shot (HyPLM report)ProteinGym DMS substitution assays at commit 144fe22 (217 assays)Spearman (Spearman correlation averaged over the 217 substitution assays of ProteinGym v1.3 with the benchmark's official aggregation, from masked-marginal zero-shot scores; per-assay values rounded to three decimals before aggregation), unitless0.4090.415ESM-2
Stability (HyPLM report)Stability (TAPE, HyPLM report)Spearman (Spearman correlation on the TAPE stability regression split), unitless0.7580.639ESM3
Thermostability (HyPLM report)Thermostability, Meltome human cell via SaProt (HyPLM report)Spearman (Spearman correlation on the Meltome human-cell split as packaged by SaProt, which the report lists separately from its own FLIP Meltome human-cell row), unitless0.6790.697ESM-2
Comparisons in pfmbench primary benchmark evidence (1)

What disagrees

Reference model
ESM-2
This study
median relative difference +0.20
Other studies
-0.06 to -0.01
Compared
ESM3, ESM-2
Study that disagrees
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design, ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction, ProteinGym AMFR assay-level Spearman comparison, ProteinGym clinical supervised substitutions AUC summary at commit 144fe22, ProteinGym clinical zero-shot indels AUC summary at commit 144fe22, ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22, ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22
Other studies
19 studiespfmbench primary benchmark evidence, HyCell CITATION.cff (commit 0dcc533c), HyCell package README (Tencent Hunyuan, commit 0dcc533c), HyCell NOTICE (commit 0dcc533c), HyCell pyproject.toml (commit 0dcc533c), HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01), HyCell weights hycell-16m weights.safetensors (commit 0dcc533c), HyCell checkpoint sidecar weights.json (commit 0dcc533c), HyInv LICENSE (MIT, commit 0dcc533c), HyInv package README (Tencent Hunyuan, commit 0dcc533c), HyInv Technical Report (Hyra Team, 2026-10-14), HyPLM-Flash engine source hyplm/engine/graphs.py (commit 0dcc533c), HyPLM-Flash engine source hyplm/engine/model.py (commit 0dcc533c), HyPLM-Flash engine source hyplm/engine/packing.py (commit 0dcc533c), HyPLM-Flash LICENSE (MIT, commit 0dcc533c), HyPLM-Flash package README (Tencent Hunyuan, commit 0dcc533c), HyPLM Technical Report (Hyra Team, 2026-10-12), Hyra Results repository LICENSE (commit 0dcc533c), Hyra Results repository README (Tencent Hunyuan, commit 0dcc533c)
Rule
Gap outlier; same task, different protocols
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (38)
StudyToolProtocol or benchmarkMetricValue
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csvESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym AMFR official zero-shot Spearman snapshotSpearman0.26
ProteinGym AMFR assay-level Spearman comparison; Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csvESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym AMFR official zero-shot Spearman snapshotSpearman0.29
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))0.4
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)0.405
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)0.202
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman)0.219
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)0.173
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)0.234
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman)0.405
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)0.317
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)0.405
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)0.387
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)0.488
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)0.467
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)0.341
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)0.422
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)0.432
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)0.467
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)0.436
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM2 (15B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)0.333
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays))0.466
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman)0.487
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman)0.337
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman)0.307
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman)0.285
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman)0.363
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman)0.43
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman)0.4
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman)0.47
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman)0.389
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman)0.641
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman)0.575
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman)0.397
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman)0.466
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman)0.48
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman)0.548
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman)0.53
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22; Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.pyESM3 open (1.4B) [ProteinGym 144fe22b snapshot]ProteinGym v1.3 zero-shot DMS substitutionsSpearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman)0.406
In the other studies (24)
StudyToolProtocol or benchmarkMetricValue
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)AAV low-vs-high (HyPLM report)Spearman (Spearman correlation on the FLIP AAV low-vs-high split)0.279
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)AAV low-vs-high (HyPLM report)Spearman (Spearman correlation on the FLIP AAV low-vs-high split)0.287
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)AAV two-vs-many (HyPLM report)Spearman (Spearman correlation on the FLIP AAV two-vs-many split)0.518
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)AAV two-vs-many (HyPLM report)Spearman (Spearman correlation on the FLIP AAV two-vs-many split)0.551
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)Amylase by mutation count (HyPLM report)Spearman (Spearman correlation on the FLIP2 amylase split partitioned by mutation count)0.575
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)Amylase by mutation count (HyPLM report)Spearman (Spearman correlation on the FLIP2 amylase split partitioned by mutation count)0.524
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)Amylase one-to-many (HyPLM report)Spearman (Spearman correlation on the FLIP2 amylase one-to-many split)0.64
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)Amylase one-to-many (HyPLM report)Spearman (Spearman correlation on the FLIP2 amylase one-to-many split)0.618
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)Fluorescence (HyPLM report)Spearman (Spearman correlation on the TAPE fluorescence regression split)0.664
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)Fluorescence (HyPLM report)Spearman (Spearman correlation on the TAPE fluorescence regression split)0.669
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)GB1 low-vs-high (HyPLM report)Spearman (Spearman correlation on the FLIP GB1 low-vs-high split)0.377
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)GB1 low-vs-high (HyPLM report)Spearman (Spearman correlation on the FLIP GB1 low-vs-high split)0.455
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)Meltome human cell (HyPLM report)Spearman (Spearman correlation on the FLIP Meltome human-cell split)0.696
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)Meltome human cell (HyPLM report)Spearman (Spearman correlation on the FLIP Meltome human-cell split)0.682
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)Meltome mixed (HyPLM report)Spearman (Spearman correlation on the FLIP Meltome mixed split)0.633
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)Meltome mixed (HyPLM report)Spearman (Spearman correlation on the FLIP Meltome mixed split)0.573
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)ProteinGym zero-shot (HyPLM report)Spearman (Spearman correlation averaged over the 217 substitution assays of ProteinGym v1.3 with the benchmark's official aggregation, from masked-marginal zero-shot scores; per-assay values rounded to three decimals before aggregation)0.415
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)ProteinGym zero-shot (HyPLM report)Spearman (Spearman correlation averaged over the 217 substitution assays of ProteinGym v1.3 with the benchmark's official aggregation, from masked-marginal zero-shot scores; per-assay values rounded to three decimals before aggregation)0.409
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)Stability (HyPLM report)Spearman (Spearman correlation on the TAPE stability regression split)0.639
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)Stability (HyPLM report)Spearman (Spearman correlation on the TAPE stability regression split)0.758
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM-2 650M (HyPLM report)Thermostability (HyPLM report)Spearman (Spearman correlation on the Meltome human-cell split as packaged by SaProt, which the report lists separately from its own FLIP Meltome human-cell row)0.697
HyPLM Technical Report (Hyra Team, 2026-10-12)ESM3-open (HyPLM report)Thermostability (HyPLM report)Spearman (Spearman correlation on the Meltome human-cell split as packaged by SaProt, which the report lists separately from its own FLIP Meltome human-cell row)0.679
pfmbench primary benchmark evidenceESM-2PFMBench PROTEINGYM-ZS: ProteinGym zero-shot variant effect predictionSpearman0.439
pfmbench primary benchmark evidenceESM3PFMBench PROTEINGYM-ZS: ProteinGym zero-shot variant effect predictionSpearman0.414

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-5079a7277df150f3 · All contested results · How findings are made and followed up