rewirebio.iobenchmarks
Result

Spearman correlation difference (HyPLM-Flash minus the baseline, Spearman correlation over the 217 ProteinGym v1.3 substitution assays with the official aggregation; 95% interval from 5,000 bootstrap replicates over assays, recomputing the official aggregation each time, with identical resamples for both models; the reference system is HyPLM-Flash) of DPLM-650M (HyPLM report) on ProteinGym DMS substitution assays at commit 144fe22 (217 assays)

−0.001 Spearman correlation difference (HyPLM-Flash minus the baseline, Spearman correlation over the 217 ProteinGym v1.3 substitution assays with the official aggregation; 95% interval from 5,000 bootstrap replicates over assays, recomputing the official aggregation each time, with identical resamples for both models; the reference system is HyPLM-Flash)

Methods

Tested configuration
DPLM-650M (HyPLM report)
Protocol
ProteinGym zero-shot paired difference from HyPLM-Flash (HyPLM report)
Dataset subset
ProteinGym DMS substitution assays at commit 144fe22 (217 assays)
Procedure
ProteinGym zero-shot paired difference from HyPLM-Flash (HyPLM report)
Evaluation
DPLM-650M on proteingym-paired-difference (HyPLM report)
Coverage
Not reported scored / Not reported eligible
Uncertainty
95% CI -0.019 to 0.015. Paired bootstrap over assays with identical resamples; the interval includes zero, which is the report's basis for calling the difference not significant.
Evidence
Author-reported evaluation · source checkedHyPLM Technical Report (Hyra Team, 2026-10-12) · Section 4.1, sentence 'Its differences to ESM-2 650M ...', parenthesis for DPLM-650M (page 4 of 13)

A source-checked result verifies the numerical transcription, not every model or protocol detail. Evaluation metadata: source checked. Source checked does not mean independently reproduced.

Reproduction

Split
No train, validation or test split: ProteinGym substitutions is scored zero-shot over all its assays, and Table 4 prints '217 assays' in place of split sizes.
Adaptation
Frozen BF16 representations; the encoder is not fine-tuned
Scoring implementation
As stated in the protocol

No execution recipe has been verified for this exact configuration and evaluation. A benchmark's general instructions may use different inputs, splits or model settings.

Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

1 evidence row matching the loaded filters

Claims, original sources and review scope · Release 2026-11-21-c8bea25e5abb
Property and statementOriginal source and locationReview and provenance
Reported result
−0.001
Individual claims
HyPLM Technical Report (Hyra Team, 2026-10-12)

Original source ↗

Section 4.1, sentence 'Its differences to ESM-2 650M ...', parenthesis for DPLM-650M (page 4 of 13)

Version: Dated 2026-10-12 on page 1; retrieved at commit 0dcc533cc9f8f76c90ecbfe944df816b6672328a (13 pages)
Retrieved: 2026-11-06T11:08:21Z

source checked

["independent-cell-check","deterministic-table-parse","pdf-text-parse","source-hash-verification"] · 2026-11-06T00:00:00Z

author reported

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.printed_value

Source artifact SHA-256: d81d88a415a91c2b00d8e97513a90c03ccffd857ec75033fa7de77227dea1710

Hash scope: pdftotext -layout text layer, parsed by extract/parse.py with every row label, metric label and cell count asserted.

Inspected artifact

Extraction artifact SHA-256: d81d88a415a91c2b00d8e97513a90c03ccffd857ec75033fa7de77227dea1710

Extraction artifact

Sources and history

Release 2026-11-21-c8bea25e5abb · Record review: source checked

1 source record and release historyDownload this release (gzip)
Technical metadata and extraction receipts

Stable ID: hyplm-flash-20261106-result-proteingym-paired-difference-dplm-650m

claim scope
Not a win. Section 4.1 states HyPLM-Flash's ProteinGym differences to ESM-2 650M, ESM C 600M, DPLM-650M and E1-600M are not significant (each 95% interval includes zero), and neither are those to ESM-2 3B, ESM-2 15B and ESM C 6B; the order of these point estimates does not show that one model is better.
metric
spearman-correlation-difference
metric direction
higher
metric qualifier
HyPLM-Flash minus the baseline, Spearman correlation over the 217 ProteinGym v1.3 substitution assays with the official aggregation; 95% interval from 5,000 bootstrap replicates over assays, recomputing the official aggregation each time, with identical resamples for both models; the reference system is HyPLM-Flash
numeric value
-0.001
printed source cell
−0.001 [−0.019, 0.015]
printed value
−0.001
review
artifact sha256: d81d88a415a91c2b00d8e97513a90c03ccffd857ec75033fa7de77227dea1710; method: independent-cell-check; deterministic-table-parse; pdf-text-parse; source-hash-verification; method note: Independent review pass. All six pinned artifacts were re-downloaded at the pinned commit and their SHA-256 checked against the source records; pdftotext -layout reproduced the archived text layer byte for byte. npm run batch -- check-values was run over the whole batch (value-check.tsv). The 469 recorded cells of Tables 3, 5, 6 and 7 were then re-read by a reviewer script written for this pass, which takes each table's column order from its printed header line and splits every body row by printed cell, without using extract/parse.py; every printed value and column matched. The two arithmetic constraints were re-derived independently rather than taken from the extractor's checker: each Table 3 perplexity against exp of its NLL over the NLL's own rounding interval, and each Table 5 average against the mean of its five categories. Column assignment in Tables 6 and 7 is anchored outside those tables, by Table 3's NLL values and by Figure 3a's and Table 5's ProteinGym values, and the four printed paired differences reconcile with the Table 6 ProteinGym row. The 15 locators check-values cannot parse, four Figure 3a bar labels and eleven prose values, were read by hand against the text layer; in Figure 3a each bar label and its value share one line. extract/check_cells.py was rerun and reproduces cell-check.tsv.; retrieval url: https://raw.githubusercontent.com/Tencent-Hunyuan/Hyra-results/0dcc533cc9f8f76c90ecbfe944df816b6672328a/AI4Science/hyplm_flash/HyPLM_Technical_Report.pdf; reviewed at: 2026-11-06T00:00:00Z; reviewer: claude; reviewer note: Claude research agent; no human review claimed. Reviewed by a different agent from the one that extracted the batch. Transcription checked against the pinned source; nothing was reproduced and no value was recomputed.
source locator
Section 4.1, sentence 'Its differences to ESM-2 650M ...', parenthesis for DPLM-650M (page 4 of 13)
uncertainty
level: 0.95; lower: -0.019; method: bootstrap_percentile; note: Paired bootstrap over assays with identical resamples; the interval includes zero, which is the report's basis for calling the difference not significant.; printed: [−0.019, 0.015]; resamples: 5000; type: confidence_interval; upper: 0.015
unit
unitless
Related records

Suggest a correction