rewirebio.iobenchmarks
Evidence claim

PFMBench against ProteinGym on ESM-2 and ESM3: ESM-2 half explained, ESM3 input asked upstream

Discrepancy review of ESM-2 · PFMBench ANTI-RES · Accuracy

investigating

Review of discrepancy-4db3e5d24233e19b, revision 3: investigating, reported upstream.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

38 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-21-c8bea25e5abb
Property and statementOriginal source and locationReview and provenance
Action
reported-upstream
Context-only references
pfmbench primary benchmark evidence

Original source ↗

expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280

Version: 2506.14796v1
Retrieved: 2026-09-16T21:06:29.716004+00:00

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.action

Source artifact SHA-256: 59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Awaiting
PFMBench's answer on which ESM3 input produced the Table 4 ProteinGym score.
Context-only references
pfmbench primary benchmark evidence

Original source ↗

expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280

Version: 2506.14796v1
Retrieved: 2026-09-16T21:06:29.716004+00:00

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.awaiting

Source artifact SHA-256: 59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Discrepancy id
discrepancy-4db3e5d24233e19b
Context-only references
pfmbench primary benchmark evidence

Original source ↗

expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280

Version: 2506.14796v1
Retrieved: 2026-09-16T21:06:29.716004+00:00

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.discrepancy_id

Source artifact SHA-256: 59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Claimed field
discrepancy_review
Context-only references
pfmbench primary benchmark evidence

Original source ↗

expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280

Version: 2506.14796v1
Retrieved: 2026-09-16T21:06:29.716004+00:00

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.field

Source artifact SHA-256: 59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Investigation url
https://github.com/rewire-bio/rewire-benchmark-data/issues/210
Context-only references
pfmbench primary benchmark evidence

Original source ↗

expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280

Version: 2506.14796v1
Retrieved: 2026-09-16T21:06:29.716004+00:00

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.investigation_url

Source artifact SHA-256: 59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Next check
2026-12-02
Context-only references
pfmbench primary benchmark evidence

Original source ↗

expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280

Version: 2506.14796v1
Retrieved: 2026-09-16T21:06:29.716004+00:00

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.next_check

Source artifact SHA-256: 59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Rationale
ESM-2 is explained by aggregation: with the same plain mean over the same 217 assays, ProteinGym's per-assay file at commit 144fe22 gives ESM-2 650M 0.438 against PFMBench's 0.439 (ESM-C 600M 0.434 against 0.434). ESM3 is not: like for like the gap is 0.414 against 0.483. PFMBench Table 4 labels ESM3's input as sequence only, while ProteinGym conditions ESM3 on the AF2 structure, which would explain the gap, but PFMBench's released code builds ESM3Model without sequence_only and always tokenises the AF2 structure, misaligned with the sequence window on the 16 targets longer than 1,022 residues. Which configuration produced 0.414 was asked upstream.
Context-only references
pfmbench primary benchmark evidence

Original source ↗

expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280

Version: 2506.14796v1
Retrieved: 2026-09-16T21:06:29.716004+00:00

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.rationale

Source artifact SHA-256: 59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Report url
https://github.com/biomap-research/PFMBench/issues/5
Context-only references
pfmbench primary benchmark evidence

Original source ↗

expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280

Version: 2506.14796v1
Retrieved: 2026-09-16T21:06:29.716004+00:00

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.report_url

Source artifact SHA-256: 59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Reported on
2026-11-18
Context-only references
pfmbench primary benchmark evidence

Original source ↗

expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280

Version: 2506.14796v1
Retrieved: 2026-09-16T21:06:29.716004+00:00

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.reported_on

Source artifact SHA-256: 59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Revision
3
Context-only references
pfmbench primary benchmark evidence

Original source ↗

expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280

Version: 2506.14796v1
Retrieved: 2026-09-16T21:06:29.716004+00:00

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.revision

Source artifact SHA-256: 59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-21-c8bea25e5abb · Record review: source checked

1 source record and release historyDownload this release (gzip)
Technical metadata and extraction receipts

Stable ID: discrepancy-review-20261119-pfmbench-esm2-esm3-r3

areas
proteins-complexes
contexts
research
field
discrepancy_review
discrepancy id
discrepancy-4db3e5d24233e19b
revision
3
value
investigating
action
reported-upstream
rationale
ESM-2 is explained by aggregation: with the same plain mean over the same 217 assays, ProteinGym's per-assay file at commit 144fe22 gives ESM-2 650M 0.438 against PFMBench's 0.439 (ESM-C 600M 0.434 against 0.434). ESM3 is not: like for like the gap is 0.414 against 0.483. PFMBench Table 4 labels ESM3's input as sequence only, while ProteinGym conditions ESM3 on the AF2 structure, which would explain the gap, but PFMBench's released code builds ESM3Model without sequence_only and always tokenises the AF2 structure, misaligned with the sequence window on the 16 targets longer than 1,022 residues. Which configuration produced 0.414 was asked upstream.
investigation url
https://github.com/rewire-bio/rewire-benchmark-data/issues/210
report url
https://github.com/biomap-research/PFMBench/issues/5
reported on
2026-11-18
awaiting
PFMBench's answer on which ESM3 input produced the Table 4 ProteinGym score.
next check
2026-12-02
source locator
expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280
review
method: ai-assisted-source-review; reviewer: claude; reviewer note: Recorded by a Claude agent; the upstream question was posted by the repository owner on 2026-11-18; no human source review claimed.; reviewed at: 2026-11-19T12:00:00Z; method note: Recomputed plain means from ProteinGym's per-assay file and read PFMBench's Table 4 and released scoring code (issue 280).; note: Records where the discrepancy stands. No result value or status was changed.
Related records

Suggest a correction