AUROC (global ROC-AUC over all variants; zero-shot) of SIFT (Brandes et al. 2023 clinical benchmarks) on ClinVar missense pathogenic vs benign, primary isoforms (Brandes et al. 2023, from the EVE portal)
0.874 AUROC (global ROC-AUC over all variants; zero-shot)
Methods
- Tested configuration
- SIFT (Brandes et al. 2023 clinical benchmarks)
- Protocol
- ClinVar missense pathogenicity, variants scored by all 19 eligible methods (Brandes et al. 2023 Fig. 2e-f, Supplementary Table 2)
- Dataset
- ClinVar missense pathogenic vs benign, primary isoforms (Brandes et al. 2023, from the EVE portal)
- Procedure
- ClinVar missense pathogenicity, variants scored by all 19 eligible methods (Brandes et al. 2023 Fig. 2e-f, Supplementary Table 2)
- Evaluation
- SIFT on ClinVar missense, variants scored by all 19 methods (Brandes et al. 2023)
- Coverage
- Not reported scored / Not reported eligible
- Uncertainty
- Not reported by the source
- Evidence
- Independent external evaluation · source checkedBrandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_ClinVar' cell B4, row 'SIFT', column 'AUC_AUROC'
A source-checked result verifies the numerical transcription, not every model or protocol detail. Evaluation metadata: source checked. Source checked does not mean independently reproduced.
Reproduction
- Split
- Variants scored by all 19 eligible methods
- Adaptation
- zero-shot
- Scoring implementation
- Global ROC-AUC; balanced PRC-AUC averaged over 100 balanced downsamples
No execution recipe has been verified for this exact configuration and evaluation. A benchmark's general instructions may use different inputs, splits or model settings.
Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
1 evidence row matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Reported result 0.874278041098027 Individual claims | Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods Supplementary Table 2 sheet 'Table_of_results_ClinVar' cell B4, row 'SIFT', column 'AUC_AUROC' Version: Version of record, Nature Genetics 55, 1512-1522 (published 2023-08-10) | source checked ["source-hash-verification","deterministic-table-parse","independent-cell-check"] · 2026-10-14 independent paper Audit detailsExtraction: Read the pinned workbook with the stdlib OOXML reader in extract/build_esm1b.py; header A1:D1, n_var, the included flag and the 19 method labels are asserted. Independent review 2026-10-14: matched. The cells use the General format, so printed_value keeps the stored decimal. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record Archive member: artifacts/brandes2023-supplementary-table-2.xlsx.gz Extraction artifact SHA-256: |
Sources and history
Release 2026-10-22-1b7cf027e4f6 · Record review: source checked
2 source records and release history
- Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Original source · Version of record, Nature Genetics 55, 1512-1522 (published 2023-08-10)
- Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023) · Original source · Version of record, Nature Genetics 55, 1512-1522 (published 2023-08-10)
Technical metadata and extraction receipts
Stable ID: gfm-guides-20261014-esm1b-result-clinvar-all-19-sift-auroc
- printed value
- 0.874278041098027
- numeric value
- 0.874278041098027
- metric
- auroc
- metric direction
- higher
- unit
- unitless
- metric qualifier
- global ROC-AUC over all variants; zero-shot
- source locator
- Supplementary Table 2 sheet 'Table_of_results_ClinVar' cell B4, row 'SIFT', column 'AUC_AUROC'
- raw xml value
- 0.874278041098027
- printed value basis
- Shortest round-trip decimal of the number stored in the workbook cell
- missing metadata
- uncertainty: reason: unreported
- review
- method: source-hash-verification; deterministic-table-parse; independent-cell-check; method note: Workbook re-downloaded from the publisher (SHA-256 matches). An independent openpyxl reader matched all 76 workbook results (stored value, metric column, row label against the configuration's reported name, n_var and the included flag); every eligible cell in both clinical sheets was found exactly once. Six cells also checked by hand.; reviewer: claude; reviewer note: Claude (Opus 5.5) research agent, independent of the extractor; no human review claimed; date: 2026-10-14; artifact sha256: 093c32e4156a801c0dd809a437c87818a4f1d04df92e037cf40222471a3e72e2; retrieval url: https://static-content.springer.com/esm/art%3A10.1038%2Fs41588-023-01465-0/MediaObjects/41588_2023_1465_MOESM3_ESM.xlsx; note: Extraction: Read the pinned workbook with the stdlib OOXML reader in extract/build_esm1b.py; header A1:D1, n_var, the included flag and the 19 method labels are asserted. Independent review 2026-10-14: matched. The cells use the General format, so printed_value keeps the stored decimal.