Evidence claim

ProteinGym against ProteinBench on ESM3 and ProteinMPNN: different tasks

Discrepancy review of ESM3 open (1.4B) AMFR Spearman

explained

Review of discrepancy-fed245c475fd08dc: explained.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

196 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-02-a7f03c9e55a5
Property and statement	Original source and location	Review and provenance
Action investigated Context-only references	Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py Original source ↗ The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.572257+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.action` Source artifact SHA-256: `0002bc1b031c02dc2d67fde53da092c8fd301415645e8e83b425b4183570eba2` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Action investigated Context-only references	Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv Original source ↗ The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.604049+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.action` Source artifact SHA-256: `a8f498011532a74aa9fe556a50555a75e928c5837d19c06a87592ae04049b308` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Action investigated Context-only references	ProteinGym AMFR assay-level Spearman comparison Original source ↗ The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-30	not individually reviewed No individual claim review recorded Audit details Field: `attributes.action` Source artifact SHA-256: `f432423b87f79ac9778dfac86e3d95be041d246bc618dc0a406b35b0b7466437` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Action investigated Context-only references	ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22 Original source ↗ The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-10-14T13:39:32Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.action` Source artifact SHA-256: `ee61a7c09efbfbd8d9a6c9355092c798032a578fa58c6f6256b5da8778555fd4` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Date 2026-10-23 Context-only references	Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py Original source ↗ The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.572257+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.date` Source artifact SHA-256: `0002bc1b031c02dc2d67fde53da092c8fd301415645e8e83b425b4183570eba2` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Date 2026-10-23 Context-only references	Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv Original source ↗ The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.604049+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.date` Source artifact SHA-256: `a8f498011532a74aa9fe556a50555a75e928c5837d19c06a87592ae04049b308` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Date 2026-10-23 Context-only references	ProteinGym AMFR assay-level Spearman comparison Original source ↗ The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-30	not individually reviewed No individual claim review recorded Audit details Field: `attributes.date` Source artifact SHA-256: `f432423b87f79ac9778dfac86e3d95be041d246bc618dc0a406b35b0b7466437` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Date 2026-10-23 Context-only references	ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22 Original source ↗ The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-10-14T13:39:32Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.date` Source artifact SHA-256: `ee61a7c09efbfbd8d9a6c9355092c798032a578fa58c6f6256b5da8778555fd4` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Discrepancy id discrepancy-fed245c475fd08dc Context-only references	Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py Original source ↗ The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.572257+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.discrepancy_id` Source artifact SHA-256: `0002bc1b031c02dc2d67fde53da092c8fd301415645e8e83b425b4183570eba2` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Discrepancy id discrepancy-fed245c475fd08dc Context-only references	Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv Original source ↗ The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.604049+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.discrepancy_id` Source artifact SHA-256: `a8f498011532a74aa9fe556a50555a75e928c5837d19c06a87592ae04049b308` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-02-a7f03c9e55a5 · Record review: source checked

4 source records and release history

Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
ProteinGym AMFR assay-level Spearman comparison · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22 · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: discrepancy-review-20261023-proteingym-esm3-proteinmpnn

areas: proteins-complexes
contexts: research
field: discrepancy_review
discrepancy id: discrepancy-fed245c475fd08dc
revision: 1
value: explained
action: investigated
date: 2026-10-23
rationale: The mirror of discrepancy-f82ed80f71c6f0bd: ProteinGym zero-shot variant effect prediction ranks ESM3 above ProteinMPNN, and the only consensus study is ProteinBench's de novo backbone sequence design. Different endpoints; both kept as reported. A two-study disagreement should not fire twice (issue 205).
investigation url: https://github.com/rewire-bio/rewire-benchmark-data/issues/205
source locator: The outlier and consensus results of discrepancy-fed245c475fd08dc as published at https://benchmarks.rewirebio.io/contested/discrepancy-fed245c475fd08dc/ (release 2026-10-23-968c8720eea6)
review: method: automated-source-review; ai-assisted-source-review; reviewer: claude; reviewer note: Recorded by a Claude agent; no human review claimed.; date: 2026-10-23; method note: Read every outlier and consensus result of the finding, with its protocol, metric, configuration and source, from the published contested-result page, and compared the tasks, settings and configurations on each side. The original papers were not re-opened.; note: Records where the discrepancy stands, not a correction to any result. No result value or status was changed.

Related records

subject: ESM3 open (1.4B) AMFR Spearman
subject: ProteinMPNN AMFR Spearman
subject: ESM3 open (1.4B) Average_Spearman (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Depth_1 (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Depth_2 (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Depth_3 (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Depth_4 (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Depth_5+ (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Function_Activity (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Function_Binding (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Function_Expression (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Function_OrganismalFitness (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Function_Stability (ProteinGym 144fe22)
subject: ESM3 open (1.4B) High_MSA_depth (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Low_MSA_depth (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Medium_MSA_depth (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Taxa_Human (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Taxa_Other_Eukaryote (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Taxa_Prokaryote (ProteinGym 144fe22)
subject: ESM3 open (1.4B) Taxa_Virus (ProteinGym 144fe22)
subject: ProteinMPNN Average_Spearman (ProteinGym 144fe22)
subject: ProteinMPNN Depth_1 (ProteinGym 144fe22)
subject: ProteinMPNN Depth_2 (ProteinGym 144fe22)
subject: ProteinMPNN Depth_3 (ProteinGym 144fe22)
subject: ProteinMPNN Depth_4 (ProteinGym 144fe22)
subject: ProteinMPNN Depth_5+ (ProteinGym 144fe22)
subject: ProteinMPNN Function_Activity (ProteinGym 144fe22)
subject: ProteinMPNN Function_Binding (ProteinGym 144fe22)
subject: ProteinMPNN Function_Expression (ProteinGym 144fe22)
subject: ProteinMPNN Function_OrganismalFitness (ProteinGym 144fe22)
subject: ProteinMPNN Function_Stability (ProteinGym 144fe22)
subject: ProteinMPNN High_MSA_depth (ProteinGym 144fe22)
subject: ProteinMPNN Low_MSA_depth (ProteinGym 144fe22)
subject: ProteinMPNN Medium_MSA_depth (ProteinGym 144fe22)
subject: ProteinMPNN Taxa_Human (ProteinGym 144fe22)
subject: ProteinMPNN Taxa_Other_Eukaryote (ProteinGym 144fe22)
subject: ProteinMPNN Taxa_Prokaryote (ProteinGym 144fe22)
subject: ProteinMPNN Taxa_Virus (ProteinGym 144fe22)

Suggest a correction