Evidence claim

PFMBench against ProteinGym on ESM-2 and ESM3: ProteinGym zero-shot scores differ

Discrepancy review of ESM-2 · PFMBench ANTI-RES · Accuracy

investigating

Review of discrepancy-a2ed36d1ca54b5b9, revision 2: investigating. Discrepancy rules version 4 gave discrepancy-4db3e5d24233e19b a new ID (two-study-conflict); the state and resolution are carried forward unchanged.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

36 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-02-a7f03c9e55a5
Property and statement	Original source and location	Review and provenance
Date 2026-10-24 Context-only references	pfmbench primary benchmark evidence Original source ↗ The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Version: 2506.14796v1 Retrieved: 2026-09-16T21:06:29.716004+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.date` Source artifact SHA-256: `59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2` Hash scope: Exact retrieved primary paper artifact bytes. Inspected artifact
Discrepancy id discrepancy-a2ed36d1ca54b5b9 Context-only references	pfmbench primary benchmark evidence Original source ↗ The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Version: 2506.14796v1 Retrieved: 2026-09-16T21:06:29.716004+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.discrepancy_id` Source artifact SHA-256: `59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2` Hash scope: Exact retrieved primary paper artifact bytes. Inspected artifact
Claimed field discrepancy_review Context-only references	pfmbench primary benchmark evidence Original source ↗ The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Version: 2506.14796v1 Retrieved: 2026-09-16T21:06:29.716004+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.field` Source artifact SHA-256: `59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2` Hash scope: Exact retrieved primary paper artifact bytes. Inspected artifact
Investigation url https://github.com/rewire-bio/rewire-benchmark-data/issues/210 Context-only references	pfmbench primary benchmark evidence Original source ↗ The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Version: 2506.14796v1 Retrieved: 2026-09-16T21:06:29.716004+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.investigation_url` Source artifact SHA-256: `59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2` Hash scope: Exact retrieved primary paper artifact bytes. Inspected artifact
Next check 2026-11-06 Context-only references	pfmbench primary benchmark evidence Original source ↗ The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Version: 2506.14796v1 Retrieved: 2026-09-16T21:06:29.716004+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.next_check` Source artifact SHA-256: `59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2` Hash scope: Exact retrieved primary paper artifact bytes. Inspected artifact
Previous ids 1 values discrepancy-4db3e5d24233e19b Context-only references	pfmbench primary benchmark evidence Original source ↗ The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Version: 2506.14796v1 Retrieved: 2026-09-16T21:06:29.716004+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.previous_ids` Source artifact SHA-256: `59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2` Hash scope: Exact retrieved primary paper artifact bytes. Inspected artifact
Rationale Eleven of PFMBench's twelve comparisons are supervised probing tasks with no counterpart elsewhere. The twelfth, PFMBench's ProteinGym zero-shot row, gives ESM3 0.414 and ESM-2 0.439, while ProteinGym's own files give ESM3 open 1.4B 0.466 and ESM-2 650M 0.414. Being checked: PFMBench's assay subset, scoring rule and checkpoints (issue 210). Context-only references	pfmbench primary benchmark evidence Original source ↗ The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Version: 2506.14796v1 Retrieved: 2026-09-16T21:06:29.716004+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.rationale` Source artifact SHA-256: `59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2` Hash scope: Exact retrieved primary paper artifact bytes. Inspected artifact
Revision 2 Context-only references	pfmbench primary benchmark evidence Original source ↗ The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Version: 2506.14796v1 Retrieved: 2026-09-16T21:06:29.716004+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.revision` Source artifact SHA-256: `59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2` Hash scope: Exact retrieved primary paper artifact bytes. Inspected artifact
Source location The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Context-only references	pfmbench primary benchmark evidence Original source ↗ The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Version: 2506.14796v1 Retrieved: 2026-09-16T21:06:29.716004+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.source_locator` Source artifact SHA-256: `59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2` Hash scope: Exact retrieved primary paper artifact bytes. Inspected artifact
Claimed value investigating Context-only references	pfmbench primary benchmark evidence Original source ↗ The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6) Version: 2506.14796v1 Retrieved: 2026-09-16T21:06:29.716004+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.value` Source artifact SHA-256: `59c7bbb888e8e91f33c1e2cabfde062c32381d6ca727d23b6655c977aabf97a2` Hash scope: Exact retrieved primary paper artifact bytes. Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-02-a7f03c9e55a5 · Record review: source checked

1 source record and release history

pfmbench primary benchmark evidence · Original source · 2506.14796v1

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: discrepancy-review-20261023-pfmbench-esm2-esm3-r2

areas: proteins-complexes
contexts: research
field: discrepancy_review
discrepancy id: discrepancy-a2ed36d1ca54b5b9
revision: 2
value: investigating
date: 2026-10-24
rationale: Eleven of PFMBench's twelve comparisons are supervised probing tasks with no counterpart elsewhere. The twelfth, PFMBench's ProteinGym zero-shot row, gives ESM3 0.414 and ESM-2 0.439, while ProteinGym's own files give ESM3 open 1.4B 0.466 and ESM-2 650M 0.414. Being checked: PFMBench's assay subset, scoring rule and checkpoints (issue 210).
investigation url: https://github.com/rewire-bio/rewire-benchmark-data/issues/210
next check: 2026-11-06
source locator: The outlier and consensus results of discrepancy-4db3e5d24233e19b as published at https://benchmarks.rewirebio.io/contested/discrepancy-4db3e5d24233e19b/ (release 2026-10-23-968c8720eea6)
review: method: automated-source-review; ai-assisted-source-review; reviewer: claude; reviewer note: Recorded by a Claude agent; no human review claimed.; date: 2026-10-24; method note: Read every outlier and consensus result of the finding, with its protocol, metric, configuration and source, from the published contested-result page, and compared the tasks, settings and configurations on each side. The original papers were not re-opened.; note: Carried forward by scripts/kg/migrate-discrepancy-ids.py when rules version 4 re-keyed the finding; the evidence and decision are those of discrepancy-review-20261023-pfmbench-esm2-esm3.
previous ids: discrepancy-4db3e5d24233e19b

Related records

subject: ESM-2 · PFMBench ANTI-RES · Accuracy
subject: ESM-2 · PFMBench BINDING-DB · Spearman
subject: ESM-2 · PFMBench CLONING-CLF · AUROC
subject: ESM-2 · PFMBench DEEPLOC2 · F1 score
subject: ESM-2 · PFMBench DEEPSOL · AUROC
subject: ESM-2 · PFMBench EC · F1 score
subject: ESM-2 · PFMBench MAT-PROD · Accuracy
subject: ESM-2 · PFMBench METAL-ION · Accuracy
subject: ESM-2 · PFMBench PDB-BIND · Spearman
subject: ESM-2 · PFMBench PROTEINGYM-ZS · Spearman
subject: ESM-2 · PFMBench SEC-STRUCT · Accuracy
subject: ESM-2 · PFMBench STABILITY · Spearman
subject: ESM3 · PFMBench ANTI-RES · Accuracy
subject: ESM3 · PFMBench BINDING-DB · Spearman
subject: ESM3 · PFMBench CLONING-CLF · AUROC
subject: ESM3 · PFMBench DEEPLOC2 · F1 score
subject: ESM3 · PFMBench DEEPSOL · AUROC
subject: ESM3 · PFMBench EC · F1 score
subject: ESM3 · PFMBench MAT-PROD · Accuracy
subject: ESM3 · PFMBench METAL-ION · Accuracy
subject: ESM3 · PFMBench PDB-BIND · Spearman
subject: ESM3 · PFMBench PROTEINGYM-ZS · Spearman
subject: ESM3 · PFMBench SEC-STRUCT · Accuracy
subject: ESM3 · PFMBench STABILITY · Spearman

Suggest a correction