Benchmark

ProteinGym

ProteinGym separates experimental variant-effect and clinical annotation tasks under supervised and zero-shot regimes.

Sources

OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation

90 evaluations · 231 results

Overview

Datasets

DMS assays and human clinical variants, with substitution and indel collections kept separate.

Metrics

Zero-shot DMS: Spearman, NDCG, AUC, MCC and top-K recall; supervised DMS: Spearman/MSE; clinical: AUC. Aggregation first groups assays by UniProt ID, then averages functional categories.

Allowed inputs

Variant and target protein sequences; comparator modalities separately include alignments, structures and function annotations.

Sources

OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation

Evaluation procedure diagram

How it worksEvaluation procedure

Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions.

Sources (2)

OATML-Markslab/ProteinGym official source; proteingym primary benchmark evidence · Pinned README: benchmark data; performance metrics and aggregation; Supervised DMS benchmarking

Source reviewed · Automated source review, 2026-09-23. All specifications and missing details

Results

Each comparison retains its reviewed evaluation scope, dataset and metric. Results are shown without a pooled ranking.

Find a protocol, dataset or metric

Evaluation scope and datasetMetric and reported setup

ProteinGym SUP-SUB-MSE-AVG: Supervised substitutions, average over splits, MSE

mse (error) · Lower values are better.

ProteinGym SUP-SUB-MSE-AVG: Supervised substitutions, average over splits, MSE · ProteinGym substitution DMS assays (ProteinGym split)

Evidence origin: Author-reported evaluation.

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design · Table 3, column(Supervised substitutions, average over splits, MSE)

MSE is better when lower; every other metric here is better when higher.
Zero-shot, supervised and indel benchmarks use different protocols and assay sets, so their figures are not comparable to each other.

Comparison details and limitations

Every method ProteinGym reports on Supervised substitutions, average over splits, MSE, scored with MSE on ProteinGym substitution DMS assays.

Author-reported numbers, source checked but not independently reproduced.

Automated source review: 2026-09-18. Numerical source review does not establish independent reproduction.

Dots show point estimates. Whiskers show only explicitly defined uncertainty (standard deviation, standard error or a labelled interval); their definitions remain in Table. Unresolved uncertainty is not plotted. Differences do not establish statistical significance.

Find a tested model or method

Showing 10 of 10 matching rows.

Zoom to observed range

Tested configuration

0.660.7660.8720.9781.08

Reported score

Methods and evaluation design

Procedure, tasks and evaluated configurations

How it works

Evaluation methodology

DMS assays and human clinical variants, with substitution and indel collections kept separate. Five-fold random, contiguous-position and modulo-position cross-validation are separate supervised DMS regimes. The original clinical analysis uses available ClinVar labels with explicit overlap warnings; zero-shot scoring does not fit on assay labels. Zero-shot DMS: Spearman, NDCG, AUC, MCC and top-K recall; supervised DMS: Spearman/MSE; clinical: AUC. Aggregation first groups assays by UniProt ID, then averages functional categories. Supervised DMS evaluations distinguish five-fold random, contiguous-position and modulo-position partitions. The original clinical benchmark explicitly warns that supervised methods may overlap ClinVar labels and that population-frequency training can leak information into benign-variant evaluation.

Sources (2)

OATML-Markslab/ProteinGym official source; proteingym primary benchmark evidence · Pinned README: benchmark data; performance metrics and aggregation; Supervised DMS benchmarking; Sections on supervised DMS and clinical benchmarking

Evaluation design

Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.

Tasks

Protocols

These source-backed links do not make different protocols or scores interchangeable.

Baseline coverage

Reference methods help show what a model adds beyond simple controls. We track a null control and a conventional method for each protocol.

1 of 6 active baseline roles have published Rewire measurements in this release. Measurements on a selected protocol do not establish coverage of an entire suite.

Baseline status by linked protocol

ProteinGym v1.3 zero-shot DMS substitutions · 0/2 roles measured
ProteinGym v1.3 AMFR seeded-random control · 1/2 roles measured
ProteinGym v1.3 AMFR substitution assay · 0/2 roles measured

Protocol coverage CSV · Model evaluation matrix · Source table · Release and checksums

Coverage is derived from release 2026-09-29-06401fd5b220. Source citations describe the original records; they do not validate an unreviewed baseline proposal. No results have been generated by this audit.

Run this benchmark

Choose a concrete protocol before running an evaluation. Its inputs, split and scoring rules determine which results can be compared.

Official run instructions

Recompute zero-shot substitution benchmark metrics from published prediction files, without running protein models.

Checked against the official instructions on 2026-09-17. These commands have not been executed by rewire. Running them does not automatically reproduce the published scores.

Before you start

A working conda environment named proteingym_env with the evaluation dependencies; merge_all_scores.sh explicitly activates that name. The inspected README does not provide a complete installation recipe.
Download and unpack the v1.3 DMS substitution data and zero-shot substitution scores from the official Resources table; update scripts/zero_shot_config.sh to their actual locations.
The reference files and config.json must agree with the chosen data/model-score release. No model checkpoint is needed for recomputing metrics from existing scores.

1. Check out the reviewed repository
Repository checkout wrapper: the detached revision selects the exact official source inspected for this guide.
```
git clone https://github.com/OATML-Markslab/ProteinGym.git
cd ProteinGym
git checkout --detach 144fe22b07dfaeec2b366f2346203a9838a55b4c
```
OATML-Markslab/ProteinGym / README.md · Pinned repository revision; README.md
2. Download the two documented resource archives
Adapted from the official download template with filenames selected from its table. Configure the unpacked paths in scripts/zero_shot_config.sh before the next step; downloading into the checkout does not automatically match its default cache paths.
```
VERSION="v1.3"
FILENAME="DMS_ProteinGym_substitutions.zip"
curl -o "${FILENAME}" "https://marks.hms.harvard.edu/proteingym/ProteinGym_${VERSION}/${FILENAME}"
unzip "${FILENAME}"
FILENAME="zero_shot_substitutions_scores.zip"
curl -o "${FILENAME}" "https://marks.hms.harvard.edu/proteingym/ProteinGym_${VERSION}/${FILENAME}"
unzip "${FILENAME}"
```
OATML-Markslab/ProteinGym / README.md; OATML-Markslab/ProteinGym / scripts/zero_shot_config.sh · README.md lines 108–138, 175–182; scripts/zero_shot_config.sh lines 9–35
3. Merge scores and calculate substitution metrics
Run from this directory: both scripts source ../zero_shot_config.sh and use ../../ paths. The README root-relative invocation alone does not establish this working directory.
```
cd scripts/scoring_DMS_zero_shot
bash merge_all_scores.sh
bash performance_substitutions.sh
```
OATML-Markslab/ProteinGym / README.md; OATML-Markslab/ProteinGym / scripts/scoring_DMS_zero_shot/merge_all_scores.sh; OATML-Markslab/ProteinGym / scripts/scoring_DMS_zero_shot/performance_substitutions.sh · README.md lines 175–182; scripts/scoring_DMS_zero_shot/merge_all_scores.sh lines 1–12; performance_substitutions.sh lines 1–11

Expected outputs

One merged score file per assay at the configured DMS_merged_score_folder_subs.
Metric and aggregate performance files under benchmarks/DMS_zero_shot/substitutions, including assay-, UniProt- and functional-category summaries.

Scope and limitations

This recomputes metrics for supplied scores; it does not reproduce the models that generated those scores.
The README download example describes predictions but names the assay-data archive; the guide uses the separate filenames in its resource table.
The README lists 4.4 GB for unpacked substitution scores while the config comment lists 31 GB. Storage requirements are therefore unresolved between these two inspected files.
The environment, remote archive bytes and file contents are not verified by execution. The inspected instructions do not establish a minimum RAM/VRAM requirement, wall-clock runtime or monetary cost; none is inferred.

Strengths, limitations and unresolved questions

Strengths and limitations

Strengths supported by sources

Protein-level and functional-category aggregation reduces over-weighting of proteins with many assays.
Sources
OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation

Limitations and conditions

ProteinGym’s DMS cross-validation controls and clinical-label overlap risks are different. Zero-shot, supervised, substitution and indel tasks must remain separate, with original assay identities preserved.
Sources
proteingym primary benchmark evidence · Sections on supervised DMS and clinical benchmarking
A fully scored assay or selected subset cannot establish complete ProteinGym coverage. The local v1.3 adapter retains original denominators and withholds suite metrics for subsets, smoke tests and incomplete predictions; it does not estimate an uncertainty interval.
Sources
rewirebench: ProteinGym guide · docs/proteingym.md lines 31–33 and 57–70

Profile review details

Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Stable record: discovery-benchmark-proteingym

Specifications

Inputs, training, access and other details

Explanatory profile: source reviewed · Automated source review, 2026-09-23. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
Property	Description and evidence
Datasets	DMS assays and human clinical variants, with substitution and indel collections kept separate. Sources OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation
Splits	In the original supervised single-substitution analysis, random folds assign mutants, contiguous folds group sequence positions into segments, and modulo folds assign positions by remainder. Five folds are usual; the paper excepts F7YBW8_MESOW and SPG1_STRSG, which have four mutated positions. The pinned supervised scorer uses random splits only for indels; substitution schemes remain separate. Zero-shot scoring does not fit assay labels. Sources (3) Protocol metadata evidence: proteingym-paper.xml; Protocol metadata evidence: proteingym-proteingym-performance_DMS_supervised_benchmarks.py; OATML-Markslab/ProteinGym official source · PMC10723403 sections S22 (DMS assays) and S45 (Cross-validation schemes); performance_DMS_supervised_benchmarks.py lines 59–64; README regime descriptions
Metrics	Zero-shot DMS: Spearman, NDCG, AUC, MCC and top-K recall; supervised DMS: Spearman/MSE; clinical: AUC. Aggregation first groups assays by UniProt ID, then averages functional categories. Sources OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation
Baselines	The README distinguishes sequence-only baselines such as ESM-1v, alignment-based approaches such as DeepSequence/EVE, and sequence-plus-structure approaches such as SaProt; clinical baselines use dbNSFP 4.4a. Sources OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation
Leakage controls	Supervised DMS evaluations distinguish five-fold random, contiguous-position and modulo-position partitions. The original clinical benchmark explicitly warns that supervised methods may overlap ClinVar labels and that population-frequency training can leak information into benign-variant evaluation. Sources proteingym primary benchmark evidence · Sections on supervised DMS and clinical benchmarking
Uncertainty	For DMS substitution summaries at revision 144fe22, the zero-shot scorer bootstraps 10,000 times within functional categories after averaging by UniProt and category, and reports the standard deviation of differences from the model with the highest aggregate metric. The supervised scorer also reports bootstrap differences, using ProteinNPT as its default reference. These errors are not absolute per-model confidence intervals or variation across model-training seeds. Sources (2) Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py; Protocol metadata evidence: proteingym-proteingym-performance_DMS_supervised_benchmarks.py · performance_DMS_benchmarks.py lines 95–111 and 296–315; performance_DMS_supervised_benchmarks.py lines 16–43 and 116–118
Entity type	Benchmark suite with separate DMS/clinical, substitution/indel and supervised/zero-shot tracks. Sources OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation
Organisms	DMS collections span taxa; the clinical track concerns human proteins. Taxa-specific performance files are provided. Sources OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation
Assays	Deep mutational scanning measurements and curated benign/pathogenic clinical annotations. Sources OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation
Allowed inputs	Variant and target protein sequences; comparator modalities separately include alignments, structures and function annotations. Sources OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation
Adaptation	Separate zero-shot scoring and supervised learning regimes; labeled-data access must follow the chosen track. Sources OATML-Markslab/ProteinGym official source · Pinned README: benchmark data; performance metrics and aggregation
Release scope	The pinned ProteinGym revision 144fe22 describes releases v1.0–v1.3. v1.3 adds zero-shot DMS substitution baselines; it is not interchangeable with the original paper release, supervised DMS or clinical tracks. Sources OATML-Markslab/ProteinGym official source · README.md lines 110–138 and 226–231; revision 144fe22b07dfaeec2b366f2346203a9838a55b4c
DMS substitution coverage	The pinned v1.3 reference lists 217 assays and 2,465,767 assay-variant records, including multiple substitutions. These are reference denominators, not a claim that a catalogued model scored the complete track. Sources Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv · DMS_substitutions.csv: all 217 data rows; sum DMS_total_number_mutants (not DMS_number_single_mutants)

Applicable tests and references

Applicability is distinct from a completed evaluation.

ESM-1v · Proposed association
ESM-2 · Proposed association

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Papers and result coverage

Last literature check: 2026-09-17. Dated primary-source discovery and protocol/table screening. Source checking does not mean experimental reproduction. Only separately extracted and independently reviewed numeric batches are publishable.

Paper or primary resource	Version	Reference
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	Primary full-text snapshot retrieved 2026-09-17; exact bytes pinned by SHA-256	Read source

Historical gaps recorded on 2026-09-17

The catalogue now holds 231 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.

complete comparable numeric result batch: Specific candidate tables and protocol boundaries are documented; no graph values or incomplete winner-only selection are converted into publishable rows.

Search and extraction details

primary protocol reviewed

Searches

ProteinGym benchmark 2023 2025 zero shot results official

Evidence locations

Original NeurIPS 2023 Tables 2–4; task definition and aggregation Methods

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

35 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-29-06401fd5b220
Property and statement	Original source and location	Review and provenance
Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims	proteingym primary benchmark evidence Original source ↗ Pinned README: benchmark data; performance metrics and aggregation; Supervised DMS benchmarking Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC10723403.1 Retrieved: 2026-09-16T10:41:16.517323+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.diagram.caption` Source artifact SHA-256: `4519641f13271bdd09b166e7d93232f22542489bc52a25b5a1628c3df8badce1` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ Pinned README: benchmark data; performance metrics and aggregation; Supervised DMS benchmarking Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.diagram.caption` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record
Diagram steps Allowed inputs: Variant and target protein sequences; comparator modalities separately include alignments, structures and function annotations. Splits: Five-fold random, contiguous-position and modulo-position cross-validation are separate supervised DMS regimes. The original clinical analysis uses available ClinVar labels with explicit overlap warnings; zero-shot scoring does not fit on assay labels. Metrics: Zero-shot DMS: Spearman, NDCG, AUC, MCC and top-K recall; supervised DMS: Spearman/MSE; clinical: AUC. Aggregation first groups assays by UniProt ID, then averages functional categories. Individual claims	proteingym primary benchmark evidence Original source ↗ Pinned README: benchmark data; performance metrics and aggregation; Supervised DMS benchmarking Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC10723403.1 Retrieved: 2026-09-16T10:41:16.517323+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.diagram.steps` Source artifact SHA-256: `4519641f13271bdd09b166e7d93232f22542489bc52a25b5a1628c3df8badce1` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Diagram steps Allowed inputs: Variant and target protein sequences; comparator modalities separately include alignments, structures and function annotations. Splits: Five-fold random, contiguous-position and modulo-position cross-validation are separate supervised DMS regimes. The original clinical analysis uses available ClinVar labels with explicit overlap warnings; zero-shot scoring does not fit on assay labels. Metrics: Zero-shot DMS: Spearman, NDCG, AUC, MCC and top-K recall; supervised DMS: Spearman/MSE; clinical: AUC. Aggregation first groups assays by UniProt ID, then averages functional categories. Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ Pinned README: benchmark data; performance metrics and aggregation; Supervised DMS benchmarking Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.diagram.steps` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record
Diagram title Evaluation procedure Individual claims	proteingym primary benchmark evidence Original source ↗ Pinned README: benchmark data; performance metrics and aggregation; Supervised DMS benchmarking Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC10723403.1 Retrieved: 2026-09-16T10:41:16.517323+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.diagram.title` Source artifact SHA-256: `4519641f13271bdd09b166e7d93232f22542489bc52a25b5a1628c3df8badce1` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Diagram title Evaluation procedure Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ Pinned README: benchmark data; performance metrics and aggregation; Supervised DMS benchmarking Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.diagram.title` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record
Datasets DMS assays and human clinical variants, with substitution and indel collections kept separate. Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ Pinned README: benchmark data; performance metrics and aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.0.value` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record
Splits In the original supervised single-substitution analysis, random folds assign mutants, contiguous folds group sequence positions into segments, and modulo folds assign positions by remainder. Five folds are usual; the paper excepts F7YBW8_MESOW and SPG1_STRSG, which have four mutated positions. The pinned supervised scorer uses random splits only for indels; substitution schemes remain separate. Zero-shot scoring does not fit assay labels. Individual claims	Protocol metadata evidence: proteingym-paper.xml Original source ↗ PMC10723403 sections S22 (DMS assays) and S45 (Cross-validation schemes); performance_DMS_supervised_benchmarks.py lines 59–64; README regime descriptions Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC10723403 retrieved 2026-09-23 Retrieved: 2026-09-23T18:43:10.260678+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.1.value` Source artifact SHA-256: `a58110e02c3efc8395c534d48630f577c3d45e90d48198b7f9989cf52600b045` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Splits In the original supervised single-substitution analysis, random folds assign mutants, contiguous folds group sequence positions into segments, and modulo folds assign positions by remainder. Five folds are usual; the paper excepts F7YBW8_MESOW and SPG1_STRSG, which have four mutated positions. The pinned supervised scorer uses random splits only for indels; substitution schemes remain separate. Zero-shot scoring does not fit assay labels. Individual claims	Protocol metadata evidence: proteingym-proteingym-performance_DMS_supervised_benchmarks.py Original source ↗ PMC10723403 sections S22 (DMS assays) and S45 (Cross-validation schemes); performance_DMS_supervised_benchmarks.py lines 59–64; README regime descriptions Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.564820+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.1.value` Source artifact SHA-256: `49b9c29fad95056eadddedb9be7d10397ba357d35f10cfb3719eacd35e6064f0` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Splits In the original supervised single-substitution analysis, random folds assign mutants, contiguous folds group sequence positions into segments, and modulo folds assign positions by remainder. Five folds are usual; the paper excepts F7YBW8_MESOW and SPG1_STRSG, which have four mutated positions. The pinned supervised scorer uses random splits only for indels; substitution schemes remain separate. Zero-shot scoring does not fit assay labels. Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ PMC10723403 sections S22 (DMS assays) and S45 (Cross-validation schemes); performance_DMS_supervised_benchmarks.py lines 59–64; README regime descriptions Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Release scope, DMS substitution coverage, Splits, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.1.value` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

View linked audit checks and correction history

Release 2026-09-29-06401fd5b220 · Record review: discovered

12 source records and release history

OATML-Markslab/ProteinGym official source · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design · Original source · Primary full-text snapshot retrieved 2026-09-17; exact bytes pinned by SHA-256
OATML-Markslab/ProteinGym / README.md · Original source · Git commit 144fe22b07dfaeec2b366f2346203a9838a55b4c
OATML-Markslab/ProteinGym / scripts/zero_shot_config.sh · Original source · Git commit 144fe22b07dfaeec2b366f2346203a9838a55b4c
OATML-Markslab/ProteinGym / scripts/scoring_DMS_zero_shot/merge_all_scores.sh · Original source · Git commit 144fe22b07dfaeec2b366f2346203a9838a55b4c
OATML-Markslab/ProteinGym / scripts/scoring_DMS_zero_shot/performance_substitutions.sh · Original source · Git commit 144fe22b07dfaeec2b366f2346203a9838a55b4c
proteingym primary benchmark evidence · Original source · PMC10723403.1
Protocol metadata evidence: proteingym-paper.xml · Original source · PMC10723403 retrieved 2026-09-23
Protocol metadata evidence: proteingym-proteingym-performance_DMS_supervised_benchmarks.py · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
rewirebench: ProteinGym guide · Original source · f80cef7f818bec33e51b7f43ad499eb5078c8d87

Download this release

Technical metadata and extraction receipts

Stable ID: discovery-benchmark-proteingym

areas: protein-function
entity level: suite
scope note: Specialist molecular or omics evaluation; protocol details require review before numerical comparison.
task: Protein variant effect prediction
version: Not reported
benchmark research: review date: 2026-09-17; status: primary_protocol_reviewed; primary sources: evidence-expansion-proteingym-a3b08cc4; inspected locators: Original NeurIPS 2023 Tables 2–4; task definition and aggregation Methods; searched queries: ProteinGym benchmark 2023 2025 zero shot results official; gaps: complete comparable numeric result batch: Specific candidate tables and protocol boundaries are documented; no graph values or incomplete winner-only selection are converted into publishable rows.; claim scope: Dated primary-source discovery and protocol/table screening. Source checking does not mean experimental reproduction. Only separately extracted and independently reviewed numeric batches are publishable.
historical missing metadata: dataset release: unextracted; metric implementation: unextracted; split manifest: unextracted; version: unextracted
metadata review scope: historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
entity classification: review date: 2026-09-17; rationale: The cited profile describes a collection of evaluation tasks or protocols; retain it as the top-level benchmark suite. Its datasets and individual protocols remain separate records.; source ids: src-discovery-oatml-markslab-proteingym; source locator: Pinned README: benchmark data; performance metrics and aggregation; ambiguities: None recorded
run guide: record id: discovery-benchmark-proteingym; summary: Recompute zero-shot substitution benchmark metrics from published prediction files, without running protein models.; status: source_reviewed_not_executed; prerequisites: A working conda environment named proteingym_env with the evaluation dependencies; merge_all_scores.sh explicitly activates that name. The inspected README does not provide a complete installation recipe.; Download and unpack the v1.3 DMS substitution data and zero-shot substitution scores from the official Resources table; update scripts/zero_shot_config.sh to their actual locations.; The reference files and config.json must agree with the chosen data/model-score release. No model checkpoint is needed for recomputing metrics from existing scores.; steps: title: Check out the reviewed repository; shell: git clone https://github.com/OATML-Markslab/ProteinGym.git cd ProteinGym git checkout --detach 144fe22b07dfaeec2b366f2346203a9838a55b4c; explanation: Repository checkout wrapper: the detached revision selects the exact official source inspected for this guide.; source ids: run-doc-proteingym-readme-md-144fe22b; source locator: Pinned repository revision; README.md; title: Download the two documented resource archives; shell: VERSION="v1.3" FILENAME="DMS_ProteinGym_substitutions.zip" curl -o "${FILENAME}" "https://marks.hms.harvard.edu/proteingym/ProteinGym_${VERSION}/${FILENAME}" unzip "${FILENAME}" FILENAME="zero_shot_substitutions_scores.zip" curl -o "${FILENAME}" "https://marks.hms.harvard.edu/proteingym/ProteinGym_${VERSION}/${FILENAME}" unzip "${FILENAME}"; explanation: Adapted from the official download template with filenames selected from its table. Configure the unpacked paths in scripts/zero_shot_config.sh before the next step; downloading into the checkout does not automatically match its default cache paths.; source ids: run-doc-proteingym-readme-md-144fe22b; run-doc-proteingym-config-sh-144fe22b; source locator: README.md lines 108–138, 175–182; scripts/zero_shot_config.sh lines 9–35; title: Merge scores and calculate substitution metrics; shell: cd scripts/scoring_DMS_zero_shot bash merge_all_scores.sh bash performance_substitutions.sh; explanation: Run from this directory: both scripts source ../zero_shot_config.sh and use ../../ paths. The README root-relative invocation alone does not establish this working directory.; source ids: run-doc-proteingym-readme-md-144fe22b; run-doc-proteingym-merge-sh-144fe22b; run-doc-proteingym-performance-sh-144fe22b; source locator: README.md lines 175–182; scripts/scoring_DMS_zero_shot/merge_all_scores.sh lines 1–12; performance_substitutions.sh lines 1–11; outputs: One merged score file per assay at the configured DMS_merged_score_folder_subs.; Metric and aggregate performance files under benchmarks/DMS_zero_shot/substitutions, including assay-, UniProt- and functional-category summaries.; limitations: This recomputes metrics for supplied scores; it does not reproduce the models that generated those scores.; The README download example describes predictions but names the assay-data archive; the guide uses the separate filenames in its resource table.; The README lists 4.4 GB for unpacked substitution scores while the config comment lists 31 GB. Storage requirements are therefore unresolved between these two inspected files.; The environment, remote archive bytes and file contents are not verified by execution. The inspected instructions do not establish a minimum RAM/VRAM requirement, wall-clock runtime or monetary cost; none is inferred.; source ids: run-doc-proteingym-readme-md-144fe22b; run-doc-proteingym-config-sh-144fe22b; run-doc-proteingym-merge-sh-144fe22b; run-doc-proteingym-performance-sh-144fe22b; review: method: official_repository_review; date: 2026-09-17
run documentation: record id: discovery-benchmark-proteingym; source ids: run-doc-proteingym-readme-md-144fe22b; run-doc-proteingym-config-sh-144fe22b; run-doc-proteingym-merge-sh-144fe22b; run-doc-proteingym-performance-sh-144fe22b; status: source_reviewed_not_executed; summary: Recompute zero-shot substitution benchmark metrics from published prediction files, without running protein models. Commands were source-reviewed only. This recomputes metrics for supplied scores; it does not reproduce the models that generated those scores.; source locator: Pinned repository revision; README.md; README.md lines 108–138, 175–182; scripts/zero_shot_config.sh lines 9–35; README.md lines 175–182; scripts/scoring_DMS_zero_shot/merge_all_scores.sh lines 1–12; performance_substitutions.sh lines 1–11

Related records

evaluates task: ProteinGym mutation effects
applicable to: ESM-1v
applicable to: ESM-2
subject: ProteinGym: evaluates task ProteinGym mutation effects
part of: ProteinGym SUP-SUB-MSE-AVG: Supervised substitutions, average over splits, MSE
part of: ProteinGym SUP-SUB-MSE-CONTIG: Supervised substitutions, contiguous split, MSE
part of: ProteinGym SUP-SUB-MSE-MOD: Supervised substitutions, modulo split, MSE
part of: ProteinGym SUP-SUB-MSE-RAND: Supervised substitutions, random split, MSE
part of: ProteinGym SUP-SUB-SPEARMAN-AVG: Supervised substitutions, average over splits, Spearman
part of: ProteinGym SUP-SUB-SPEARMAN-CONTIG: Supervised substitutions, contiguous split, Spearman
part of: ProteinGym SUP-SUB-SPEARMAN-MOD: Supervised substitutions, modulo split, Spearman
part of: ProteinGym SUP-SUB-SPEARMAN-RAND: Supervised substitutions, random split, Spearman
part of: ProteinGym ZS-INDEL-AUC: Zero-shot indels, all assays, AUC
part of: ProteinGym ZS-INDEL-SPEARMAN-ALL: Zero-shot indels, all assays, Spearman
part of: ProteinGym ZS-INDEL-SPEARMAN-DESIGNED: Zero-shot indels, designed or natural assays, Spearman
part of: ProteinGym ZS-INDEL-SPEARMAN-LIBRARY: Zero-shot indels, library assays, Spearman
part of: ProteinGym ZS-SUB-AUC: Zero-shot substitutions, AUC
part of: ProteinGym ZS-SUB-MCC: Zero-shot substitutions, MCC
part of: ProteinGym ZS-SUB-NDCG: Zero-shot substitutions, NDCG@10%
part of: ProteinGym ZS-SUB-RECALL: Zero-shot substitutions, top 10% recall
part of: ProteinGym ZS-SUB-SPEARMAN: Zero-shot substitutions, Spearman
part of: ProteinGym v1.3 zero-shot DMS substitutions
part of: ProteinGym v1.3 AMFR seeded-random control
part of: ProteinGym v1.3 AMFR substitution assay

Suggest a correction