AUROC (ROC-AUC, pathogenic versus benign ClinVar stop-gains not expected to undergo NMD (50 bp rule); zero-shot) of Number of residues lost (stop-gain baseline) on ClinVar stop-gains not expected to undergo NMD (Brandes et al. 2023)
0.65 AUROC (ROC-AUC, pathogenic versus benign ClinVar stop-gains not expected to undergo NMD (50 bp rule); zero-shot)
Methods
- Tested method
- Number of residues lost (stop-gain baseline)
- Protocol
- ClinVar stop-gain pathogenicity by NMD status, ROC-AUC (Brandes et al. 2023 Fig. 6d, Supplementary Table 2)
- Dataset subset
- ClinVar stop-gains not expected to undergo NMD (Brandes et al. 2023)
- Procedure
- ClinVar stop-gain pathogenicity by NMD status, ROC-AUC (Brandes et al. 2023 Fig. 6d, Supplementary Table 2)
- Evaluation
- num. AA lost on stop-gains not expected to undergo NMD (50 bp rule) (Brandes et al. 2023)
- Coverage
- Not reported scored / Not reported eligible
- Uncertainty
- Not reported by the source
- Evidence
- Author-reported evaluation · source checkedBrandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_stop_gains' cell B3, row 'num. AA lost', column 'AUC_no_NMD'
A source-checked result verifies the numerical transcription, not every model or protocol detail. Evaluation metadata: source checked. Source checked does not mean independently reproduced.
Reproduction
- Split
- Whole labelled set
- Adaptation
- zero-shot
- Scoring implementation
- ROC-AUC
No execution recipe has been verified for this exact configuration and evaluation. A benchmark's general instructions may use different inputs, splits or model settings.
Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
1 evidence row matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Reported result 0.649893290649593 Individual claims | Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods Supplementary Table 2 sheet 'Table_of_results_stop_gains' cell B3, row 'num. AA lost', column 'AUC_no_NMD' Version: Version of record, Nature Genetics 55, 1512-1522 (published 2023-08-10) | source checked ["source-hash-verification","deterministic-table-parse","independent-cell-check"] · 2026-10-14 author reported Audit detailsExtraction: Read the pinned workbook with the stdlib OOXML reader in extract/build_supp2.py; header row and the 4 method labels in column A asserted; 'NaN' cells skipped. Independent review 2026-10-14: matched. The cells use the General format, so printed_value keeps the stored decimal. The value is an exact whole-set ROC-AUC fraction for the Methods class counts (see the protocol limitations). Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record Archive member: artifacts/brandes2023-supplementary-table-2.xlsx.gz Extraction artifact SHA-256: |
Sources and history
Release 2026-10-22-1b7cf027e4f6 · Record review: source checked
2 source records and release history
- Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Original source · Version of record, Nature Genetics 55, 1512-1522 (published 2023-08-10)
- Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023) · Original source · Version of record, Nature Genetics 55, 1512-1522 (published 2023-08-10)
Technical metadata and extraction receipts
Stable ID: gfm-guides-20261014-esm1b-result-clinvar-stop-gains-no-nmd-residues-lost-count-auroc
- printed value
- 0.649893290649593
- numeric value
- 0.649893290649593
- metric
- auroc
- metric direction
- higher
- unit
- unitless
- metric qualifier
- ROC-AUC, pathogenic versus benign ClinVar stop-gains not expected to undergo NMD (50 bp rule); zero-shot
- source locator
- Supplementary Table 2 sheet 'Table_of_results_stop_gains' cell B3, row 'num. AA lost', column 'AUC_no_NMD'
- raw xml value
- 0.649893290649593
- printed value basis
- Shortest round-trip decimal of the number stored in the workbook cell
- missing metadata
- uncertainty: reason: unreported
- review
- method: source-hash-verification; deterministic-table-parse; independent-cell-check; method note: Workbook re-downloaded from the publisher (SHA-256 matches). An independent openpyxl reader matched all 386 results with the cell each locator names (stored value, header, row label, metric) and checked, for head-to-head cells, the system (ESM1b or the comparator whose reported name is the row label), origin, the included flag and note against the comparator configuration, and head2head_n_variants against the subset's scored_count; every numeric cell in the head-to-head columns and the indel and stop-gain sheets was found exactly once. Six cells also checked by hand.; reviewer: claude; reviewer note: Claude (Opus 5.5) research agent, independent of the extractor; no human review claimed; date: 2026-10-14; artifact sha256: 093c32e4156a801c0dd809a437c87818a4f1d04df92e037cf40222471a3e72e2; retrieval url: https://static-content.springer.com/esm/art%3A10.1038%2Fs41588-023-01465-0/MediaObjects/41588_2023_1465_MOESM3_ESM.xlsx; note: Extraction: Read the pinned workbook with the stdlib OOXML reader in extract/build_supp2.py; header row and the 4 method labels in column A asserted; 'NaN' cells skipped. Independent review 2026-10-14: matched. The cells use the General format, so printed_value keeps the stored decimal. The value is an exact whole-set ROC-AUC fraction for the Methods class counts (see the protocol limitations).