ESM3 relative to ESM-2: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design is outside the range of other studies
- Low severity
- Not yet reviewed
- Gap outlier
- Same task, different protocols
On ProteinGym mutation effects, ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design reports ESM3 at a median relative difference of +0.20 from ESM-2; other studies report -0.06 to -0.01.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
In 2 papers, ESM3 scores between 1% and 6% below ESM-2. In ProteinGym, ESM3 scores 20% above ESM-2.
| Paper | Role | Ranks ESM3 higher | Ranks ESM-2 higher | Ties | Gap to ESM-2 |
|---|---|---|---|---|---|
| ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and 6 more records · DOI | Disagrees | 18 | 0 | 1 | +20% |
| HyCell CITATION.cff (commit 0dcc533c) and 17 more records · Repository | Agrees | 4 | 6 | 1 | −1% |
| pfmbench primary benchmark evidence · DOI | Agrees | 0 | 1 | 0 | −6% |
Comparisons in ProteinGym (19)
| Protocol | Dataset | Metric | ESM3 | ESM-2 | Higher |
|---|---|---|---|---|---|
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Activity assays: mean over proteins of assay-level Spearman), unitless | 0.43 | 0.405 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Binding assays: mean over proteins of assay-level Spearman), unitless | 0.4 | 0.317 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Expression assays: mean over proteins of assay-level Spearman), unitless | 0.47 | 0.405 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, High MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.575 | 0.467 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Human proteins: mean over proteins of assay-level Spearman), unitless | 0.48 | 0.432 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Low MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.397 | 0.341 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Medium MSA depth proteins: mean over proteins of assay-level Spearman), unitless | 0.466 | 0.422 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Organismal fitness assays: mean over proteins of assay-level Spearman), unitless | 0.389 | 0.387 | Tie |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Other eukaryote proteins: mean over proteins of assay-level Spearman), unitless | 0.548 | 0.467 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Prokaryote proteins: mean over proteins of assay-level Spearman), unitless | 0.53 | 0.436 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Stability assays: mean over proteins of assay-level Spearman), unitless | 0.641 | 0.488 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, Virus proteins: mean over proteins of assay-level Spearman), unitless | 0.406 | 0.333 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays)), unitless | 0.466 | 0.4 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 1 substitution: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.487 | 0.405 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 2 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.337 | 0.202 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 3 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.307 | 0.219 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 4 substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.285 | 0.173 | ESM3 |
| ProteinGym v1.3 zero-shot DMS substitutions | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (ProteinGym zero-shot DMS substitutions, variants with 5 or more substitutions: mean over protein and selection-type groups of assay-level Spearman), unitless | 0.363 | 0.234 | ESM3 |
| ProteinGym AMFR official zero-shot Spearman snapshot | ProteinGym AMFR 4G3O official snapshot cohort | Spearman, unitless | 0.29 | 0.26 | ESM3 |
Comparisons in HyCell CITATION.cff (11)
| Protocol | Dataset | Metric | ESM3 | ESM-2 | Higher |
|---|---|---|---|---|---|
| AAV low-vs-high (HyPLM report) | FLIP AAV, low-vs-high split (FLIP split) | Spearman (Spearman correlation on the FLIP AAV low-vs-high split), unitless | 0.287 | 0.279 | ESM3 |
| AAV two-vs-many (HyPLM report) | FLIP AAV, 2-vs-rest split (FLIP split) | Spearman (Spearman correlation on the FLIP AAV two-vs-many split), unitless | 0.551 | 0.518 | ESM3 |
| Amylase by mutation count (HyPLM report) | Amylase by mutation count (FLIP2, HyPLM report) | Spearman (Spearman correlation on the FLIP2 amylase split partitioned by mutation count), unitless | 0.524 | 0.575 | ESM-2 |
| Amylase one-to-many (HyPLM report) | Amylase one-to-many (FLIP2, HyPLM report) | Spearman (Spearman correlation on the FLIP2 amylase one-to-many split), unitless | 0.618 | 0.64 | ESM-2 |
| Fluorescence (HyPLM report) | Fluorescence (TAPE) | Spearman (Spearman correlation on the TAPE fluorescence regression split), unitless | 0.669 | 0.664 | Tie |
| GB1 low-vs-high (HyPLM report) | FLIP GB1, low-vs-high split (FLIP split) | Spearman (Spearman correlation on the FLIP GB1 low-vs-high split), unitless | 0.455 | 0.377 | ESM3 |
| Meltome human cell (HyPLM report) | FLIP Thermostability, Human-Cell split (FLIP split) | Spearman (Spearman correlation on the FLIP Meltome human-cell split), unitless | 0.682 | 0.696 | ESM-2 |
| Meltome mixed (HyPLM report) | FLIP Thermostability, Mixed split (FLIP split) | Spearman (Spearman correlation on the FLIP Meltome mixed split), unitless | 0.573 | 0.633 | ESM-2 |
| ProteinGym zero-shot (HyPLM report) | ProteinGym DMS substitution assays at commit 144fe22 (217 assays) | Spearman (Spearman correlation averaged over the 217 substitution assays of ProteinGym v1.3 with the benchmark's official aggregation, from masked-marginal zero-shot scores; per-assay values rounded to three decimals before aggregation), unitless | 0.409 | 0.415 | ESM-2 |
| Stability (HyPLM report) | Stability (TAPE, HyPLM report) | Spearman (Spearman correlation on the TAPE stability regression split), unitless | 0.758 | 0.639 | ESM3 |
| Thermostability (HyPLM report) | Thermostability, Meltome human cell via SaProt (HyPLM report) | Spearman (Spearman correlation on the Meltome human-cell split as packaged by SaProt, which the report lists separately from its own FLIP Meltome human-cell row), unitless | 0.679 | 0.697 | ESM-2 |
Comparisons in pfmbench primary benchmark evidence (1)
| Protocol | Dataset | Metric | ESM3 | ESM-2 | Higher |
|---|---|---|---|---|---|
| PFMBench PROTEINGYM-ZS: ProteinGym zero-shot variant effect prediction | ProteinGym (PFMBench split) | Spearman, unitless | 0.414 | 0.439 | ESM-2 |
What disagrees
- Reference model
- ESM-2
- This study
- median relative difference +0.20
- Other studies
- -0.06 to -0.01
- Compared
- ESM3, ESM-2
- Study that disagrees
- ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design, ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction, ProteinGym AMFR assay-level Spearman comparison, ProteinGym clinical supervised substitutions AUC summary at commit 144fe22, ProteinGym clinical zero-shot indels AUC summary at commit 144fe22, ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22, ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22
- Other studies
19 studies
pfmbench primary benchmark evidence, HyCell CITATION.cff (commit 0dcc533c), HyCell package README (Tencent Hunyuan, commit 0dcc533c), HyCell NOTICE (commit 0dcc533c), HyCell pyproject.toml (commit 0dcc533c), HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01), HyCell weights hycell-16m weights.safetensors (commit 0dcc533c), HyCell checkpoint sidecar weights.json (commit 0dcc533c), HyInv LICENSE (MIT, commit 0dcc533c), HyInv package README (Tencent Hunyuan, commit 0dcc533c), HyInv Technical Report (Hyra Team, 2026-10-14), HyPLM-Flash engine source hyplm/engine/graphs.py (commit 0dcc533c), HyPLM-Flash engine source hyplm/engine/model.py (commit 0dcc533c), HyPLM-Flash engine source hyplm/engine/packing.py (commit 0dcc533c), HyPLM-Flash LICENSE (MIT, commit 0dcc533c), HyPLM-Flash package README (Tencent Hunyuan, commit 0dcc533c), HyPLM Technical Report (Hyra Team, 2026-10-12), Hyra Results repository LICENSE (commit 0dcc533c), Hyra Results repository README (Tencent Hunyuan, commit 0dcc533c)- Rule
- Gap outlier; same task, different protocols
- Model identity
- Matched across studies partly through links not yet reviewed
The compared numbers
Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-5079a7277df150f3 · All contested results · How findings are made and followed up