PFMBench against ProteinGym on ESM-2 and ESM3: ESM-2 half explained, ESM3 input asked upstream
Discrepancy review of ESM-2 · PFMBench ANTI-RES · Accuracy
investigating
Review of discrepancy-4db3e5d24233e19b, revision 3: investigating, reported upstream.
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
38 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Action reported-upstream Context-only references | pfmbench primary benchmark evidence expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280 Version: 2506.14796v1 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Exact retrieved primary paper artifact bytes. |
| Awaiting PFMBench's answer on which ESM3 input produced the Table 4 ProteinGym score. Context-only references | pfmbench primary benchmark evidence expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280 Version: 2506.14796v1 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Exact retrieved primary paper artifact bytes. |
| Discrepancy id discrepancy-4db3e5d24233e19b Context-only references | pfmbench primary benchmark evidence expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280 Version: 2506.14796v1 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Exact retrieved primary paper artifact bytes. |
| Claimed field discrepancy_review Context-only references | pfmbench primary benchmark evidence expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280 Version: 2506.14796v1 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Exact retrieved primary paper artifact bytes. |
| Investigation url https://github.com/rewire-bio/rewire-benchmark-data/issues/210 Context-only references | pfmbench primary benchmark evidence expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280 Version: 2506.14796v1 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Exact retrieved primary paper artifact bytes. |
| Next check 2026-12-02 Context-only references | pfmbench primary benchmark evidence expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280 Version: 2506.14796v1 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Exact retrieved primary paper artifact bytes. |
| Rationale ESM-2 is explained by aggregation: with the same plain mean over the same 217 assays, ProteinGym's per-assay file at commit 144fe22 gives ESM-2 650M 0.438 against PFMBench's 0.439 (ESM-C 600M 0.434 against 0.434). ESM3 is not: like for like the gap is 0.414 against 0.483. PFMBench Table 4 labels ESM3's input as sequence only, while ProteinGym conditions ESM3 on the AF2 structure, which would explain the gap, but PFMBench's released code builds ESM3Model without sequence_only and always tokenises the AF2 structure, misaligned with the sequence window on the 16 targets longer than 1,022 residues. Which configuration produced 0.414 was asked upstream. Context-only references | pfmbench primary benchmark evidence expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280 Version: 2506.14796v1 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Exact retrieved primary paper artifact bytes. |
| Report url https://github.com/biomap-research/PFMBench/issues/5 Context-only references | pfmbench primary benchmark evidence expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280 Version: 2506.14796v1 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Exact retrieved primary paper artifact bytes. |
| Reported on 2026-11-18 Context-only references | pfmbench primary benchmark evidence expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280 Version: 2506.14796v1 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Exact retrieved primary paper artifact bytes. |
| Revision 3 Context-only references | pfmbench primary benchmark evidence expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280 Version: 2506.14796v1 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Exact retrieved primary paper artifact bytes. |
Sources and history
Release 2026-11-21-c8bea25e5abb · Record review: source checked
1 source record and release history
- pfmbench primary benchmark evidence · Original source · 2506.14796v1
Technical metadata and extraction receipts
Stable ID: discrepancy-review-20261119-pfmbench-esm2-esm3-r3
- areas
- proteins-complexes
- contexts
- research
- field
- discrepancy_review
- discrepancy id
- discrepancy-4db3e5d24233e19b
- revision
- 3
- value
- investigating
- action
- reported-upstream
- rationale
- ESM-2 is explained by aggregation: with the same plain mean over the same 217 assays, ProteinGym's per-assay file at commit 144fe22 gives ESM-2 650M 0.438 against PFMBench's 0.439 (ESM-C 600M 0.434 against 0.434). ESM3 is not: like for like the gap is 0.414 against 0.483. PFMBench Table 4 labels ESM3's input as sequence only, while ProteinGym conditions ESM3 on the AF2 structure, which would explain the gap, but PFMBench's released code builds ESM3Model without sequence_only and always tokenises the AF2 structure, misaligned with the sequence window on the 16 targets longer than 1,022 residues. Which configuration produced 0.414 was asked upstream.
- investigation url
- https://github.com/rewire-bio/rewire-benchmark-data/issues/210
- report url
- https://github.com/biomap-research/PFMBench/issues/5
- reported on
- 2026-11-18
- awaiting
- PFMBench's answer on which ESM3 input produced the Table 4 ProteinGym score.
- next check
- 2026-12-02
- source locator
- expansion-p3-pfmbench: Table 4 (ProteinGym zero-shot) and the released code (zeroshot/model_interface.py on_test_epoch_end; ESM3Model construct_batch); ProteinGym DMS_substitutions_Spearman_DMS_level.csv and scoring_ESM3_substitutions.sh at commit 144fe22, as checked on rewire-bio/rewire-benchmark-data#280
- review
- method: ai-assisted-source-review; reviewer: claude; reviewer note: Recorded by a Claude agent; the upstream question was posted by the repository owner on 2026-11-18; no human source review claimed.; reviewed at: 2026-11-19T12:00:00Z; method note: Recomputed plain means from ProteinGym's per-assay file and read PFMBench's Table 4 and released scoring code (issue 280).; note: Records where the discrepancy stands. No result value or status was changed.
Related records
- subject: ESM-2 · PFMBench ANTI-RES · Accuracy
- subject: ESM-2 · PFMBench BINDING-DB · Spearman
- subject: ESM-2 · PFMBench CLONING-CLF · AUROC
- subject: ESM-2 · PFMBench DEEPLOC2 · F1 score
- subject: ESM-2 · PFMBench DEEPSOL · AUROC
- subject: ESM-2 · PFMBench EC · F1 score
- subject: ESM-2 · PFMBench MAT-PROD · Accuracy
- subject: ESM-2 · PFMBench METAL-ION · Accuracy
- subject: ESM-2 · PFMBench PDB-BIND · Spearman
- subject: ESM-2 · PFMBench PROTEINGYM-ZS · Spearman
- subject: ESM-2 · PFMBench SEC-STRUCT · Accuracy
- subject: ESM-2 · PFMBench STABILITY · Spearman
- subject: ESM3 · PFMBench ANTI-RES · Accuracy
- subject: ESM3 · PFMBench BINDING-DB · Spearman
- subject: ESM3 · PFMBench CLONING-CLF · AUROC
- subject: ESM3 · PFMBench DEEPLOC2 · F1 score
- subject: ESM3 · PFMBench DEEPSOL · AUROC
- subject: ESM3 · PFMBench EC · F1 score
- subject: ESM3 · PFMBench MAT-PROD · Accuracy
- subject: ESM3 · PFMBench METAL-ION · Accuracy
- subject: ESM3 · PFMBench PDB-BIND · Spearman
- subject: ESM3 · PFMBench PROTEINGYM-ZS · Spearman
- subject: ESM3 · PFMBench SEC-STRUCT · Accuracy
- subject: ESM3 · PFMBench STABILITY · Spearman