Evidence claim

ProteinGym repository against ProteinGym paper on ESM-1v and ESM-2: different ESM-2 sizes

Discrepancy review of ESM-1v (ensemble) AMFR Spearman

false-positive

Review of discrepancy-6e0a1543c1c68a64: false positive.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

184 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-02-a7f03c9e55a5
Property and statement	Original source and location	Review and provenance
Action investigated Context-only references	Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py Original source ↗ The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.572257+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.action` Source artifact SHA-256: `0002bc1b031c02dc2d67fde53da092c8fd301415645e8e83b425b4183570eba2` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Action investigated Context-only references	Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv Original source ↗ The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.604049+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.action` Source artifact SHA-256: `a8f498011532a74aa9fe556a50555a75e928c5837d19c06a87592ae04049b308` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Action investigated Context-only references	ProteinGym AMFR assay-level Spearman comparison Original source ↗ The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-30	not individually reviewed No individual claim review recorded Audit details Field: `attributes.action` Source artifact SHA-256: `f432423b87f79ac9778dfac86e3d95be041d246bc618dc0a406b35b0b7466437` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Action investigated Context-only references	ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22 Original source ↗ The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-10-14T13:39:32Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.action` Source artifact SHA-256: `ee61a7c09efbfbd8d9a6c9355092c798032a578fa58c6f6256b5da8778555fd4` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Date 2026-10-23 Context-only references	Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py Original source ↗ The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.572257+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.date` Source artifact SHA-256: `0002bc1b031c02dc2d67fde53da092c8fd301415645e8e83b425b4183570eba2` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Date 2026-10-23 Context-only references	Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv Original source ↗ The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.604049+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.date` Source artifact SHA-256: `a8f498011532a74aa9fe556a50555a75e928c5837d19c06a87592ae04049b308` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Date 2026-10-23 Context-only references	ProteinGym AMFR assay-level Spearman comparison Original source ↗ The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-30	not individually reviewed No individual claim review recorded Audit details Field: `attributes.date` Source artifact SHA-256: `f432423b87f79ac9778dfac86e3d95be041d246bc618dc0a406b35b0b7466437` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Date 2026-10-23 Context-only references	ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22 Original source ↗ The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-10-14T13:39:32Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.date` Source artifact SHA-256: `ee61a7c09efbfbd8d9a6c9355092c798032a578fa58c6f6256b5da8778555fd4` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Discrepancy id discrepancy-6e0a1543c1c68a64 Context-only references	Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py Original source ↗ The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.572257+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.discrepancy_id` Source artifact SHA-256: `0002bc1b031c02dc2d67fde53da092c8fd301415645e8e83b425b4183570eba2` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Discrepancy id discrepancy-6e0a1543c1c68a64 Context-only references	Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv Original source ↗ The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-23T18:39:35.604049+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.discrepancy_id` Source artifact SHA-256: `a8f498011532a74aa9fe556a50555a75e928c5837d19c06a87592ae04049b308` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-02-a7f03c9e55a5 · Record review: source checked

4 source records and release history

Protocol metadata evidence: proteingym-proteingym-performance_DMS_benchmarks.py · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
Protocol metadata evidence: proteingym-reference_files-DMS_substitutions.csv · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
ProteinGym AMFR assay-level Spearman comparison · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22 · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: discrepancy-review-20261023-proteingym-esm1v-esm2

areas: proteins-complexes
contexts: research
field: discrepancy_review
discrepancy id: discrepancy-6e0a1543c1c68a64
revision: 1
value: false-positive
action: investigated
date: 2026-10-23
rationale: Both sides are ProteinGym. The repository files at commit 144fe22 list ESM-2 at 150M, 650M, 3B and 15B, and ESM-2 650M edges ESM-1v (0.414 against 0.407). The paper table lists only ESM-2 15B, which trails ESM-1v (0.401 against 0.407). The rule compared different configurations as one system, and split one benchmark into two studies (issues 206 and 207).
investigation url: https://github.com/rewire-bio/rewire-benchmark-data/issues/206
source locator: The outlier and consensus results of discrepancy-6e0a1543c1c68a64 as published at https://benchmarks.rewirebio.io/contested/discrepancy-6e0a1543c1c68a64/ (release 2026-10-23-968c8720eea6)
review: method: automated-source-review; ai-assisted-source-review; reviewer: claude; reviewer note: Recorded by a Claude agent; no human review claimed.; date: 2026-10-23; method note: Read every outlier and consensus result of the finding, with its protocol, metric, configuration and source, from the published contested-result page, and compared the tasks, settings and configurations on each side. The original papers were not re-opened.; note: Records where the discrepancy stands, not a correction to any result. No result value or status was changed.

Related records

subject: ESM-1v (ensemble) AMFR Spearman
subject: ESM2 (150M) AMFR Spearman
subject: ESM-1v (ensemble) Average_Spearman (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Depth_1 (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Depth_2 (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Depth_4 (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Depth_5+ (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Function_Activity (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Function_Binding (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Function_Expression (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Function_Stability (ProteinGym 144fe22)
subject: ESM-1v (ensemble) High_MSA_depth (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Low_MSA_depth (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Medium_MSA_depth (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Taxa_Other_Eukaryote (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Taxa_Prokaryote (ProteinGym 144fe22)
subject: ESM-1v (ensemble) Taxa_Virus (ProteinGym 144fe22)
subject: ESM-1v (single) Depth_3 (ProteinGym 144fe22)
subject: ESM2 (15B) Depth_3 (ProteinGym 144fe22)
subject: ESM2 (15B) Depth_4 (ProteinGym 144fe22)
subject: ESM2 (15B) Depth_5+ (ProteinGym 144fe22)
subject: ESM2 (15B) Low_MSA_depth (ProteinGym 144fe22)
subject: ESM2 (15B) Taxa_Virus (ProteinGym 144fe22)
subject: ESM2 (3B) Medium_MSA_depth (ProteinGym 144fe22)
subject: ESM2 (3B) Taxa_Prokaryote (ProteinGym 144fe22)
subject: ESM2 (650M) Average_Spearman (ProteinGym 144fe22)
subject: ESM2 (650M) Depth_1 (ProteinGym 144fe22)
subject: ESM2 (650M) Depth_2 (ProteinGym 144fe22)
subject: ESM2 (650M) Function_Activity (ProteinGym 144fe22)
subject: ESM2 (650M) Function_Binding (ProteinGym 144fe22)
subject: ESM2 (650M) Function_Expression (ProteinGym 144fe22)
subject: ESM2 (650M) Function_Stability (ProteinGym 144fe22)
subject: ESM2 (650M) High_MSA_depth (ProteinGym 144fe22)
subject: ESM2 (650M) Taxa_Other_Eukaryote (ProteinGym 144fe22)
subject: ESM2 (650M) Taxa_Prokaryote (ProteinGym 144fe22)

Suggest a correction