Task

ProteinGym mutation effects

ProteinGym mutation-effect evaluation compares variant scores with measurements from individual functional assays.

Sources

OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation

No reviewed evaluations are linked here in this release. See the sources and separately identified configurations below.

0 evaluations · 0 results

Overview

Datasets

Select a particular ProteinGym release, assay subset and evaluation track; the suite contains separate substitution and indel resources.

Metrics

DMS zero-shot tracks include Spearman, NDCG, AUC, MCC and top-k recall; supervised tracks include Spearman and MSE. Choose the track-specific definition.

Allowed inputs

Protein variants, permitted sequence or structure information, and the selected assay’s reference measurements.

Sources

OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation

Evaluation procedure diagram

How it worksConceptual evaluation workflow

Conceptual task guide. Dataset preparation, parameters and scoring must come from a separately identified protocol.

Sources

OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation

limited source coverage · Automated source review, 2026-09-23. All specifications and missing details

Results

All evaluations

0 evaluations · 0 results. Different protocols are not a single leaderboard.

Applied filters: All linked evaluations

No evaluations linked in this release.

Source checking is not independent reproduction. Release 2026-09-29-06401fd5b220.

Methods and evaluation design

Procedure, tasks and evaluated configurations

How it works

Choosing an evaluation

Select the suite release, assay subset and zero-shot or supervised track. ProteinGym reports within-assay metrics and further aggregation by protein and functional category. Use the published aggregation rules; a simple mean across all assays is not automatically the suite’s reported score. Preserve the distinction between molecular assay effects and clinical labels.

Sources

OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation

Task scope

This is a task guide, not a single versioned benchmark protocol. The connected resources provide examples or concrete procedures. A candidate method or proposed control is not evidence that an evaluation has been completed.

Sources

OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation

Local assay scope

The preserved ESM-2 AMFR report covers all 2,972 variants of one selected assay, with one complete assay out of the 217-assay reference. Its scope is subset and its protocol status is partial_track; suite metrics are empty. Hashing the local assay file is recorded separately from independent verification of official archive bytes. This report does not establish full-track coverage or independent reproduction.

Sources

ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · report.json: scope; completion; protocol_results.status, complete_assays, total_assays, metrics, per_assay, data_verification and uncertainty; independently_reproduced and review_status

Evaluation design

Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.

Benchmarks

ProteinGym

These source-backed links do not make different protocols or scores interchangeable.

Run instructions

No runnable recipe has been reviewed for this task. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.

A task describes a biological question. Choose a linked protocol to obtain concrete split and scoring instructions.

Strengths, limitations and unresolved questions

Strengths and limitations

Strengths and considerations

Per-assay reporting exposes how performance varies across proteins and measurement types.
Sources
OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation

Limitations and conditions

Assays measure different properties and have different coverage. A pooled ranking can hide those differences and depends on the aggregation rule.
Sources
OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation

Profile review details

Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Stable record: catalog-task-proteingym-effects

Specifications

Inputs, training, access and other details

Explanatory profile: limited source coverage · Automated source review, 2026-09-23. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
Property	Description and evidence
Entity type	Task guide; concrete protocol identities remain separate. Sources OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation
Datasets	Select a particular ProteinGym release, assay subset and evaluation track; the suite contains separate substitution and indel resources. Sources OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation
Organisms	No shared organism population is defined at this guide level. Record it for each selected dataset. · Not applicable Sources OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation
Assays	Deep mutational scanning measurements for molecular effects; assay metadata and functional categories remain distinct. Sources OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation
Splits	No executable split is attached to this task identity. Use the selected protocol’s split manifest. · Not applicable Sources OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation
Allowed inputs	Protein variants, permitted sequence or structure information, and the selected assay’s reference measurements. Sources OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation
Adaptation	No common fitting regime is imposed here. Keep pretrained, frozen, probed, fine-tuned and conventional methods distinct where applicable. · Not applicable Sources OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation
Metrics	DMS zero-shot tracks include Spearman, NDCG, AUC, MCC and top-k recall; supervised tracks include Spearman and MSE. Choose the track-specific definition. Sources OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation
Baselines	ProteinGym supplies single-sequence, alignment-based and other comparator scores; their extra information and supervision must remain visible. Sources OATML-Markslab/ProteinGym official source · README: Overview, Results, benchmark baselines and contribution notes on aggregation
Coverage reporting	Keep release, track, selected assays, complete-assay count and original variant denominators together. The local v1.3 zero-shot adapter distinguishes a fully scored selected assay from all 217 reference assays and withholds suite metrics for subset or smoke runs. Sources rewirebench: ProteinGym guide · docs/proteingym.md lines 7–11, 31–33 and 59–63
Uncertainty	Uncertainty must be declared by the selected protocol. ProteinGym’s pinned DMS scorer reports bootstrap differences relative to a reference model; the local v1.3 adapter does not estimate an absolute performance interval. Sources (3) Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py; Protocol metadata evidence: proteingym-proteingym-performance_DMS_supervised_benchmarks.py; rewirebench: ProteinGym guide · performance_DMS_benchmarks.py lines 95–111 and 296–315; performance_DMS_supervised_benchmarks.py lines 16–43; docs/proteingym.md line 63

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Papers and result coverage

Last literature check: 2026-09-17. Dated primary-source discovery and protocol/table screening. Source checking does not mean experimental reproduction. Only separately extracted and independently reviewed numeric batches are publishable.

Paper or primary resource	Version	Reference
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design	Primary full-text snapshot retrieved 2026-09-17; exact bytes pinned by SHA-256	Read source

Historical gaps recorded on 2026-09-17

complete comparable numeric result batch: Specific candidate tables and protocol boundaries are documented; no graph values or incomplete winner-only selection are converted into publishable rows.

Search and extraction details

primary protocol reviewed

Searches

ProteinGym benchmark 2023 2025 zero shot results official

Evidence locations

Original ProteinGym Tables 2–4

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

22 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-29-06401fd5b220
Property and statement	Original source and location	Review and provenance
Diagram caption Conceptual task guide. Dataset preparation, parameters and scoring must come from a separately identified protocol. Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.diagram.caption` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record
Diagram steps Pin release, track and assays Score the eligible variants Compute per-assay metrics Apply the declared aggregation Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.diagram.steps` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record
Diagram title Conceptual evaluation workflow Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.diagram.title` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record
Entity type Task guide; concrete protocol identities remain separate. Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.0.value` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record
Datasets Select a particular ProteinGym release, assay subset and evaluation track; the suite contains separate substitution and indel resources. Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.1.value` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record
Uncertainty Uncertainty must be declared by the selected protocol. ProteinGym’s pinned DMS scorer reports bootstrap differences relative to a reference model; the local v1.3 adapter does not estimate an absolute performance interval. Individual claims	Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py Original source ↗ performance_DMS_benchmarks.py lines 95–111 and 296–315; performance_DMS_supervised_benchmarks.py lines 16–43; docs/proteingym.md line 63 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.572257+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.10.value` Source artifact SHA-256: `0002bc1b031c02dc2d67fde53da092c8fd301415645e8e83b425b4183570eba2` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Uncertainty Uncertainty must be declared by the selected protocol. ProteinGym’s pinned DMS scorer reports bootstrap differences relative to a reference model; the local v1.3 adapter does not estimate an absolute performance interval. Individual claims	Protocol metadata evidence: proteingym-proteingym-performance_DMS_supervised_benchmarks.py Original source ↗ performance_DMS_benchmarks.py lines 95–111 and 296–315; performance_DMS_supervised_benchmarks.py lines 16–43; docs/proteingym.md line 63 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.564820+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.10.value` Source artifact SHA-256: `49b9c29fad95056eadddedb9be7d10397ba357d35f10cfb3719eacd35e6064f0` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Uncertainty Uncertainty must be declared by the selected protocol. ProteinGym’s pinned DMS scorer reports bootstrap differences relative to a reference model; the local v1.3 adapter does not estimate an absolute performance interval. Individual claims	rewirebench: ProteinGym guide Original source ↗ performance_DMS_benchmarks.py lines 95–111 and 296–315; performance_DMS_supervised_benchmarks.py lines 16–43; docs/proteingym.md line 63 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87 Retrieved: 2026-09-17	source checked automated source review · 2026-09-23 Audit details Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.10.value` Source artifact SHA-256: `d2b1fc009acc590b82ca125d886f3d52b07c2f36bbe0e9cb0d30776618166b5f` Hash scope: complete file bytes Format: text
Organisms No shared organism population is defined at this guide level. Record it for each selected dataset. Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	inapplicable automated source review · 2026-09-23 Audit details Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.2.value` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record
Assays Deep mutational scanning measurements for molecular effects; assay metadata and functional categories remain distinct. Individual claims	OATML-Markslab/ProteinGym official source Original source ↗ README: Overview, Results, benchmark baselines and contribution notes on aggregation Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-16T10:30:21.811694+00:00	source checked automated source review · 2026-09-23 Audit details Follow-up review of Coverage reporting, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: `attributes.profile.facts.3.value` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

View linked audit checks and correction history

Release 2026-09-29-06401fd5b220 · Record review: discovered

9 source records and release history

MIMIC official resource · Original source · Not reported
ESM-2 official resource · Original source · Not reported
ProteinMPNN official resource · Original source · Not reported
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design · Original source · Primary full-text snapshot retrieved 2026-09-17; exact bytes pinned by SHA-256
OATML-Markslab/ProteinGym official source · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
ESM-2 8M masked-marginal scoring: local execution report (20 September 2026) · Original source · ca73fa47136d182f2d4ddb083d084712198fc0e2
rewirebench: ProteinGym guide · Original source · f80cef7f818bec33e51b7f43ad499eb5078c8d87
Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
Protocol metadata evidence: proteingym-proteingym-performance_DMS_supervised_benchmarks.py · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c

Download this release

Technical metadata and extraction receipts

Stable ID: catalog-task-proteingym-effects

areas: proteins-complexes
entity level: task
version: Not reported
task: ProteinGym mutation effects
scope note: Rank substitution effects within held-out deep-mutational-scanning assays.
benchmark research: review date: 2026-09-17; status: primary_protocol_reviewed; primary sources: evidence-expansion-proteingym-a3b08cc4; inspected locators: Original ProteinGym Tables 2–4; searched queries: ProteinGym benchmark 2023 2025 zero shot results official; gaps: complete comparable numeric result batch: Specific candidate tables and protocol boundaries are documented; no graph values or incomplete winner-only selection are converted into publishable rows.; claim scope: Dated primary-source discovery and protocol/table screening. Source checking does not mean experimental reproduction. Only separately extracted and independently reviewed numeric batches are publishable.
historical missing metadata: protocol version: not_yet_extracted
metadata review scope: historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
legacy kinds: benchmark
entity classification: review date: 2026-09-17; rationale: This record identifies the biological prediction question or a suite-specific task, rather than a uniquely fixed evaluated procedure. Preserve its task identity and leave split, model adaptation and scoring details on linked protocols/evaluations.; source ids: src-discovery-oatml-markslab-proteingym; source locator: README: Overview, Results, benchmark baselines and contribution notes on aggregation; ambiguities: None recorded

Related records

evaluates task: ProteinGym

Suggest a correction