AUROC (global ROC-AUC over the variants both methods score; zero-shot) of REVEL (Brandes et al. 2023 head-to-head comparison) on HGMD/gnomAD missense variants scored by both ESM1b and REVEL (Brandes et al. 2023)
0.973 AUROC (global ROC-AUC over the variants both methods score; zero-shot)
Methods
- Tested configuration
- REVEL (Brandes et al. 2023 head-to-head comparison)
- Protocol
- HGMD/gnomAD missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)
- Dataset subset
- HGMD/gnomAD missense variants scored by both ESM1b and REVEL (Brandes et al. 2023)
- Procedure
- HGMD/gnomAD missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)
- Evaluation
- REVEL on HGMD/gnomAD missense, head to head against ESM1b (Brandes et al. 2023)
- Coverage
- Not reported scored / Not reported eligible
- Uncertainty
- Not reported by the source
- Evidence
- Independent external evaluation · source checkedBrandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_HGMD' cell E44, row 'REVEL', column 'head2head_VEP_auc_AUROC'
A source-checked result verifies the numerical transcription, not every model or protocol detail. Evaluation metadata: source checked. Source checked does not mean independently reproduced.
Reproduction
- Split
- Variants scored by both ESM1b and REVEL
- Adaptation
- zero-shot
- Scoring implementation
- Global ROC-AUC; balanced PRC-AUC averaged over 100 balanced downsamples
No execution recipe has been verified for this exact configuration and evaluation. A benchmark's general instructions may use different inputs, splits or model settings.
Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
1 evidence row matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Reported result 0.97325258459459 Individual claims | Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods Supplementary Table 2 sheet 'Table_of_results_HGMD' cell E44, row 'REVEL', column 'head2head_VEP_auc_AUROC' Version: Version of record, Nature Genetics 55, 1512-1522 (published 2023-08-10) | source checked ["source-hash-verification","deterministic-table-parse","independent-cell-check"] · 2026-10-14 independent paper Audit detailsExtraction: Read the pinned workbook with the stdlib OOXML reader in extract/build_supp2.py; header row A1:K1, the 'Contents' column definitions, the included flag of every row and the set of 45 comparator labels asserted. Independent review 2026-10-14: matched. The cells use the General format, so printed_value keeps the stored decimal. Comparator excluded from the paper's main comparison is flagged in its configuration's training_overlap where applicable. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record Archive member: artifacts/brandes2023-supplementary-table-2.xlsx.gz Extraction artifact SHA-256: |
Sources and history
Release 2026-10-22-1b7cf027e4f6 · Record review: source checked
2 source records and release history
- Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Original source · Version of record, Nature Genetics 55, 1512-1522 (published 2023-08-10)
- Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023) · Original source · Version of record, Nature Genetics 55, 1512-1522 (published 2023-08-10)
Technical metadata and extraction receipts
Stable ID: gfm-guides-20261014-esm1b-result-hgmd-h2h-revel-vep-auroc
- printed value
- 0.97325258459459
- numeric value
- 0.97325258459459
- metric
- auroc
- metric direction
- higher
- unit
- unitless
- metric qualifier
- global ROC-AUC over the variants both methods score; zero-shot
- source locator
- Supplementary Table 2 sheet 'Table_of_results_HGMD' cell E44, row 'REVEL', column 'head2head_VEP_auc_AUROC'
- raw xml value
- 0.97325258459459
- printed value basis
- Shortest round-trip decimal of the number stored in the workbook cell
- missing metadata
- uncertainty: reason: unreported
- review
- method: source-hash-verification; deterministic-table-parse; independent-cell-check; method note: Workbook re-downloaded from the publisher (SHA-256 matches). An independent openpyxl reader matched all 386 results with the cell each locator names (stored value, header, row label, metric) and checked, for head-to-head cells, the system (ESM1b or the comparator whose reported name is the row label), origin, the included flag and note against the comparator configuration, and head2head_n_variants against the subset's scored_count; every numeric cell in the head-to-head columns and the indel and stop-gain sheets was found exactly once. Six cells also checked by hand.; reviewer: claude; reviewer note: Claude (Opus 5.5) research agent, independent of the extractor; no human review claimed; date: 2026-10-14; artifact sha256: 093c32e4156a801c0dd809a437c87818a4f1d04df92e037cf40222471a3e72e2; retrieval url: https://static-content.springer.com/esm/art%3A10.1038%2Fs41588-023-01465-0/MediaObjects/41588_2023_1465_MOESM3_ESM.xlsx; note: Extraction: Read the pinned workbook with the stdlib OOXML reader in extract/build_supp2.py; header row A1:K1, the 'Contents' column definitions, the included flag of every row and the set of 45 comparator labels asserted. Independent review 2026-10-14: matched. The cells use the General format, so printed_value keeps the stored decimal. Comparator excluded from the paper's main comparison is flagged in its configuration's training_overlap where applicable.