Datasets
Stability-labelled substitution assays within the ProteinGym DMS collection.
Task
Zero-shot substitution mutation effects: stability
The stability subset evaluates zero-shot prediction of substitution effects using experimentally measured variant phenotypes.
Sources
ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78Overview
Metrics
Spearman correlation is central; the suite also defines binarized and ranking metrics. Aggregate suite scores first group related functions.
Allowed inputs
Variant/target protein sequences, with method-specific evolutionary or structural resources.
Sources
ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78Evaluation procedure diagram
How it worksComputational evaluation flow
Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.
Sources
ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78limited source coverage · Automated source review, 2026-09-16. All specifications and missing details
Results
Results are available, but no reviewed comparison panel is linked in this release.
All evaluations
2 evaluations · 2 results. Different protocols are not a single leaderboard.
Filter evaluations
Applied filters: All linked evaluations
| Tested configuration | Protocol and dataset | Finding | Evidence and details |
|---|---|---|---|
| Configuration: ESM-2 | Task: Zero-shot substitution mutation effects: stability Dataset: ProteinGym substitution DMS: stability assays | 0.488 Mean Spearman rho unitless · unknown Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceESM-2: Zero-shot substitution mutation effects: stability Zero-shot mutation scores; average Spearman across stability-category assays. Aggregation: Not reported ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Table A7, ESM-2 (15B) row, Stability column |
| Configuration: ProteinMPNN | Task: Zero-shot substitution mutation effects: stability Dataset: ProteinGym substitution DMS: stability assays | 0.566 Mean Spearman rho unitless · unknown Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceProteinMPNN: Zero-shot substitution mutation effects: stability Zero-shot mutation scores; average Spearman across stability-category assays. Aggregation: Not reported ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Table A7, ProteinMPNN row, Stability column |
Source checking is not independent reproduction. Release 2026-09-29-06401fd5b220.
Methods and evaluation design
Procedure, tasks and evaluated configurations
How it works
Evaluation methodology
Stability-labelled substitution assays within the ProteinGym DMS collection. Zero-shot evaluation does not train on the evaluated assay labels; supervised mutation splits are a separate regime. Spearman correlation is central; the suite also defines binarized and ranking metrics. Aggregate suite scores first group related functions. For the zero-shot stability category, Table A7 includes site-independent frequencies, WaveNet, EVmutation, DeepSequence, EVE and GEMME; protein language models; hybrids; and inverse-folding models such as ESM-IF1, MIF-ST and ProteinMPNN. The supervised category has separate embedding and one-hot baselines, including one-hot without augmentation, in Table A12. Their label access and structural inputs differ. Absence of assay labels during adaptation does not establish absence of sequence families from pretraining.
Sources
ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78; Appendix Tables A7 and A12; Evaluation frameworkRecorded evaluations
Each evaluation records what was tested and under which conditions.
Run instructions
No runnable recipe has been reviewed for this task. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.
A task describes a biological question. Choose a linked protocol to obtain concrete split and scoring instructions.
Strengths, limitations and unresolved questions
Strengths and limitations
Strengths and considerations
No source-reviewed explanatory claims are recorded here yet.
Limitations and conditions
- The stability subgroup and the equal-functional-group suite average are different aggregates. Sequence-only, alignment-based and structure-dependent comparators do not receive identical information.
Sources
ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Appendix Tables A7 and A12; Evaluation framework
Profile review details
Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.
Stable record: reported-task-6243658a1bc215Specifications
Inputs, training, access and other details
Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Datasets | Stability-labelled substitution assays within the ProteinGym DMS collection.SourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 |
| Splits | Zero-shot evaluation does not train on the evaluated assay labels; supervised mutation splits are a separate regime.SourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 |
| Metrics | Spearman correlation is central; the suite also defines binarized and ranking metrics. Aggregate suite scores first group related functions.SourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 |
| Baselines | For the zero-shot stability category, Table A7 includes site-independent frequencies, WaveNet, EVmutation, DeepSequence, EVE and GEMME; protein language models; hybrids; and inverse-folding models such as ESM-IF1, MIF-ST and ProteinMPNN. The supervised category has separate embedding and one-hot baselines, including one-hot without augmentation, in Table A12. Their label access and structural inputs differ.SourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Appendix Tables A7 and A12; Evaluation framework |
| Leakage controls | Absence of assay labels during adaptation does not establish absence of sequence families from pretraining.SourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 |
| Uncertainty | The cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim. · Not reported in inspected sourcesSourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 |
| Entity type | Paper-specific computational evaluation protocol.SourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 |
| Organisms | Proteins represented in the stability subset; exact taxon membership is assay-specific.SourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 |
| Assays | DMS substitution stability measurements.SourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 |
| Allowed inputs | Variant/target protein sequences, with method-specific evolutionary or structural resources.SourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 |
| Adaptation | Zero-shot evaluation without fitting on the evaluated assay labels.SourcesProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 |
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Papers and result coverage
Last literature check: 2026-09-17. Primary-paper discovery and source inspection. Source-checked results are not independently reproduced experiments.
| Paper or primary resource | Version | Reference |
|---|---|---|
| ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction | PMC10723403.1 | Read source DOI: 10.1101/2023.12.07.570727 |
Historical gaps recorded on 2026-09-17
The catalogue now holds 2 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.
- Other functional-category columns do not belong to the stability benchmark.
- This pinned 2023 paper table is not the latest ProteinGym leaderboard; do not overwrite later-version results.
- MSA and structure inputs differ across model categories; no universal sequence-only ranking.
Search and extraction details
primary comparison table screened
Searches
- "PMC10723403"
Evidence locations
- Table A7 and caption
- Benchmark methodology: zero-shot DMS
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
17 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Diagram caption Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol. Individual claims | ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 Version: PMC10723403.1 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Diagram steps
| ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 Version: PMC10723403.1 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Computational evaluation flow Individual claims | ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 Version: PMC10723403.1 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Datasets Stability-labelled substitution assays within the ProteinGym DMS collection. Individual claims | ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 Version: PMC10723403.1 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits Zero-shot evaluation does not train on the evaluated assay labels; supervised mutation splits are a separate regime. Individual claims | ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 Version: PMC10723403.1 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Adaptation Zero-shot evaluation without fitting on the evaluated assay labels. Individual claims | ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 Version: PMC10723403.1 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Metrics Spearman correlation is central; the suite also defines binarized and ranking metrics. Aggregate suite scores first group related functions. Individual claims | ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 Version: PMC10723403.1 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Baselines For the zero-shot stability category, Table A7 includes site-independent frequencies, WaveNet, EVmutation, DeepSequence, EVE and GEMME; protein language models; hybrids; and inverse-folding models such as ESM-IF1, MIF-ST and ProteinMPNN. The supervised category has separate embedding and one-hot baselines, including one-hot without augmentation, in Table A12. Their label access and structural inputs differ. Individual claims | ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction Appendix Tables A7 and A12; Evaluation framework Version: PMC10723403.1 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Leakage controls Absence of assay labels during adaptation does not establish absence of sequence families from pretraining. Individual claims | ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 Version: PMC10723403.1 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Uncertainty The cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim. Individual claims | ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78 Version: PMC10723403.1 | unreported automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
View linked audit checks and correction history
Release 2026-09-29-06401fd5b220 · Record review: needs review
2 source records and release history
- ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Original source · PMC10723403.1
- ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction · Original source · PMC10723403.1
Technical metadata and extraction receipts
Stable ID: reported-task-6243658a1bc215
- areas
- proteins-complexes
- tasks
- Zero-shot substitution mutation effects: stability
- entity level
- task
- version
- Not reported
- task
- Zero-shot substitution mutation effects: stability
- scope note
- Paper-specific evaluation task; protocol completeness requires further extraction.
- benchmark research
- review date: 2026-09-17; status: primary_comparison_table_screened; primary sources: expansion-p3-proteingym-2023; inspected locators: Table A7 and caption; Benchmark methodology: zero-shot DMS; searched queries: "PMC10723403"; gaps: Other functional-category columns do not belong to the stability benchmark.; This pinned 2023 paper table is not the latest ProteinGym leaderboard; do not overwrite later-version results.; MSA and structure inputs differ across model categories; no universal sequence-only ranking.; claim scope: Primary-paper discovery and source inspection. Source-checked results are not independently reproduced experiments.
- historical missing metadata
- protocol version: not_reported_in_legacy_extract; split: not_reported_in_legacy_extract
- metadata review scope
- historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
- legacy kinds
- benchmark
- entity classification
- review date: 2026-09-17; rationale: This source-scoped record identifies the biological prediction task and holds its paper context. Preserve the existing task identity; exact split, model adaptation and scoring remain in linked evaluations or separate protocol records.; source ids: proteingym-2023; source locator: Methods: zero-shot DMS evaluation and supervised protocols; cached text lines 54, 67, 78; ambiguities: A paper- or suite-specific task may constrain some inputs or metrics; that alone does not make it interchangeable with a complete versioned protocol. No protocol equivalence is inferred.; Some legacy profile Entity type facts use the generic phrase computational evaluation protocol. That boilerplate is not sufficient to establish a single fixed protocol identity or to merge this task with another protocol record.