rewirebio.iobenchmarks
Protocol

ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)

Global ROC-AUC of each indel score for pathogenic versus benign ClinVar in-frame indels, over all indels and by indel type.

16 evaluations · 16 results

Overview

An explanatory profile has not yet been reviewed for this record.

Consult the linked sources for architecture or protocol details. Missing evidence is not evidence of a missing capability.

16 recorded evaluations, 16 metric rows. A comparison chart has not yet been validated for these results. The table retains the individual findings and their sources.

View coverage and remaining gaps across all benchmarks

Results

Results are available, but no reviewed comparison panel is linked in this release.

All evaluations

16 evaluations · 16 results. Different protocols are not a single leaderboard.

Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)

Sorted by AUROC (ROC-AUC, pathogenic versus benign ClinVar all in-frame indels; zero-shot) (higher is better). The best value in each column is highlighted. Decimals are rounded for display; each value links to the printed value and its source.

Tested configurationProtocol and datasetAUROC (ROC-AUC, pathogenic versus benign ClinVar all in-frame indels; zero-shot)AUROC (ROC-AUC, pathogenic versus benign ClinVar deletion-insertions; zero-shot)AUROC (ROC-AUC, pathogenic versus benign ClinVar deletions; zero-shot)AUROC (ROC-AUC, pathogenic versus benign ClinVar insertions; zero-shot)
ESM1b absolute-valued PLLR indel score (Brandes et al. 2023)EvaluationClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)0.874 (best in column)Not reportedNot reportedNot reported
ESM1b PLLR indel score (Brandes et al. 2023)EvaluationClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)0.869Not reportedNot reportedNot reported
ESM1b weighted PLLR indel score (Brandes et al. 2023)EvaluationClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)0.861Not reportedNot reportedNot reported
CADD in-frame indel scores (Brandes et al. 2023)EvaluationClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)0.836Not reportedNot reportedNot reported
Pairwise alignment score (indel baseline)EvaluationClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)0.586Not reportedNot reportedNot reported
BlastP score (indel baseline)EvaluationClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)0.582Not reportedNot reportedNot reported
LevenshteinEvaluationClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)0.502Not reportedNot reportedNot reported
CADD in-frame indel scores (Brandes et al. 2023)EvaluationClinVar in-frame deletions (Brandes et al. 2023)Not reportedNot reported0.848Not reported
ESM1b absolute-valued PLLR indel score (Brandes et al. 2023)EvaluationClinVar in-frame deletions (Brandes et al. 2023)Not reportedNot reported0.875 (best in column)Not reported
ESM1b PLLR indel score (Brandes et al. 2023)EvaluationClinVar in-frame deletions (Brandes et al. 2023)Not reportedNot reported0.869Not reported
CADD in-frame indel scores (Brandes et al. 2023)EvaluationClinVar in-frame deletion-insertions (Brandes et al. 2023)Not reported0.671Not reportedNot reported
ESM1b absolute-valued PLLR indel score (Brandes et al. 2023)EvaluationClinVar in-frame deletion-insertions (Brandes et al. 2023)Not reported0.887 (best in column)Not reportedNot reported
ESM1b PLLR indel score (Brandes et al. 2023)EvaluationClinVar in-frame deletion-insertions (Brandes et al. 2023)Not reported0.857Not reportedNot reported
CADD in-frame indel scores (Brandes et al. 2023)EvaluationClinVar in-frame insertions (Brandes et al. 2023)Not reportedNot reportedNot reported0.757
ESM1b absolute-valued PLLR indel score (Brandes et al. 2023)EvaluationClinVar in-frame insertions (Brandes et al. 2023)Not reportedNot reportedNot reported0.831
ESM1b PLLR indel score (Brandes et al. 2023)EvaluationClinVar in-frame insertions (Brandes et al. 2023)Not reportedNot reportedNot reported0.859 (best in column)
All 16 result rows with coverage, uncertainty and sources
Filter evaluations

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configurationProtocol and datasetFindingEvidence and details
Method: BlastP score (indel baseline)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset: ClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)
0.582 AUROC (ROC-AUC, pathogenic versus benign ClinVar all in-frame indels; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

BlastP on all in-frame indels (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell B7, row 'BlastP', column 'AUC_indels'
Configuration: CADD in-frame indel scores (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset: ClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)
0.836 AUROC (ROC-AUC, pathogenic versus benign ClinVar all in-frame indels; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Independent external evaluation · Source checked
Methods, coverage and source

CADD on all in-frame indels (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell B5, row 'CADD', column 'AUC_indels'
Configuration: CADD in-frame indel scores (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset subset: ClinVar in-frame deletions (Brandes et al. 2023)
0.848 AUROC (ROC-AUC, pathogenic versus benign ClinVar deletions; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Independent external evaluation · Source checked
Methods, coverage and source

CADD on deletions (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell D5, row 'CADD', column 'AUC_del'
Configuration: ESM1b absolute-valued PLLR indel score (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset subset: ClinVar in-frame deletions (Brandes et al. 2023)
0.875 AUROC (ROC-AUC, pathogenic versus benign ClinVar deletions; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

ESM1b-aPLLR on deletions (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell D2, row 'ESM1b-aPLLR', column 'AUC_del'
Configuration: ESM1b PLLR indel score (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset subset: ClinVar in-frame deletions (Brandes et al. 2023)
0.869 AUROC (ROC-AUC, pathogenic versus benign ClinVar deletions; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

ESM1b-PLLR on deletions (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell D3, row 'ESM1b-PLLR', column 'AUC_del'
Configuration: CADD in-frame indel scores (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset subset: ClinVar in-frame deletion-insertions (Brandes et al. 2023)
0.671 AUROC (ROC-AUC, pathogenic versus benign ClinVar deletion-insertions; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Independent external evaluation · Source checked
Methods, coverage and source

CADD on deletion-insertions (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell E5, row 'CADD', column 'AUC_delins'
Configuration: ESM1b absolute-valued PLLR indel score (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset subset: ClinVar in-frame deletion-insertions (Brandes et al. 2023)
0.887 AUROC (ROC-AUC, pathogenic versus benign ClinVar deletion-insertions; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

ESM1b-aPLLR on deletion-insertions (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell E2, row 'ESM1b-aPLLR', column 'AUC_delins'
Configuration: ESM1b PLLR indel score (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset subset: ClinVar in-frame deletion-insertions (Brandes et al. 2023)
0.857 AUROC (ROC-AUC, pathogenic versus benign ClinVar deletion-insertions; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

ESM1b-PLLR on deletion-insertions (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell E3, row 'ESM1b-PLLR', column 'AUC_delins'
Configuration: ESM1b absolute-valued PLLR indel score (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset: ClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)
0.874 AUROC (ROC-AUC, pathogenic versus benign ClinVar all in-frame indels; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

ESM1b-aPLLR on all in-frame indels (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell B2, row 'ESM1b-aPLLR', column 'AUC_indels'
Configuration: ESM1b PLLR indel score (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset: ClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)
0.869 AUROC (ROC-AUC, pathogenic versus benign ClinVar all in-frame indels; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

ESM1b-PLLR on all in-frame indels (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell B3, row 'ESM1b-PLLR', column 'AUC_indels'
Configuration: ESM1b weighted PLLR indel score (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset: ClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)
0.861 AUROC (ROC-AUC, pathogenic versus benign ClinVar all in-frame indels; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

ESM1b-wPLLR on all in-frame indels (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell B4, row 'ESM1b-wPLLR', column 'AUC_indels'
Configuration: CADD in-frame indel scores (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset subset: ClinVar in-frame insertions (Brandes et al. 2023)
0.757 AUROC (ROC-AUC, pathogenic versus benign ClinVar insertions; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Independent external evaluation · Source checked
Methods, coverage and source

CADD on insertions (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell C5, row 'CADD', column 'AUC_ins'
Configuration: ESM1b absolute-valued PLLR indel score (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset subset: ClinVar in-frame insertions (Brandes et al. 2023)
0.831 AUROC (ROC-AUC, pathogenic versus benign ClinVar insertions; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

ESM1b-aPLLR on insertions (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell C2, row 'ESM1b-aPLLR', column 'AUC_ins'
Configuration: ESM1b PLLR indel score (Brandes et al. 2023)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset subset: ClinVar in-frame insertions (Brandes et al. 2023)
0.859 AUROC (ROC-AUC, pathogenic versus benign ClinVar insertions; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

ESM1b-PLLR on insertions (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell C3, row 'ESM1b-PLLR', column 'AUC_ins'
Method: LevenshteinProtocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset: ClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)
0.502 AUROC (ROC-AUC, pathogenic versus benign ClinVar all in-frame indels; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

Edit distance on all in-frame indels (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell B8, row 'Edit distance', column 'AUC_indels'
Method: Pairwise alignment score (indel baseline)Protocol: ClinVar in-frame indel pathogenicity, ROC-AUC (Brandes et al. 2023 Fig. 6b, Supplementary Table 2)
Dataset: ClinVar in-frame indels, pathogenic vs benign (Brandes et al. 2023)
0.586 AUROC (ROC-AUC, pathogenic versus benign ClinVar all in-frame indels; zero-shot)
unitless · higher

Uncertainty: Not reported by the source

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

Alignment on all in-frame indels (Brandes et al. 2023)

gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

Aggregation: Not reported

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · Supplementary Table 2 sheet 'Table_of_results_indels' cell B6, row 'Alignment', column 'AUC_indels'

Source checking is not independent reproduction. Release 2026-10-22-1b7cf027e4f6.

Methods and evaluation design

Procedure, tasks and evaluated configurations

Recorded evaluations

Each evaluation records what was tested and under which conditions.

Explore all linked results

Baseline coverage

Reference methods help show what a model adds beyond simple controls. We track a null control and a conventional method for each protocol.

0 of 2 active baseline roles have published Rewire measurements in this release. Measurements on a selected protocol do not establish coverage of an entire suite.

No execution recipe linked to this protocol. Recipe availability does not establish a completed evaluation.

Author-reported evaluations
12
External evaluations
4

Literature evidence is not a Rewire measurement. Executed but unpublished runs and private review status are not included.

Null control

Proposed control: requires review

Select a task-valid null control after reviewing inputs and metric

Protocol-specific applicability, permitted inputs, access, split, evaluator and execution requirements need review before implementation or execution.

This is a suggested selection rule, not a validated method or a measured score.

Conventional reference

Proposed control: requires review

Select an upstream conventional reference after reviewing the full protocol

Protocol-specific applicability, permitted inputs, access, split, evaluator and execution requirements need review before implementation or execution.

This is a suggested selection rule, not a validated method or a measured score.

Protocol coverage CSV (gzip) · Model evaluation matrix (gzip) · Source table (gzip) · Release and checksums (gzip)

Coverage is derived from release 2026-10-22-1b7cf027e4f6. Source citations describe the original records; they do not validate an unreviewed baseline proposal. No results have been generated by this audit.

Run instructions

No runnable recipe has been reviewed for this protocol. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.

Strengths, limitations and unresolved questions

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

0 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-22-1b7cf027e4f6
Property and statementOriginal source and locationReview and provenance

No evidence rows match these filters. Choose another scope or clear the search.

Sources and history

Release 2026-10-22-1b7cf027e4f6 · Record review: source checked

2 source records and release historyDownload this release (gzip)
Technical metadata and extraction receipts

Stable ID: gfm-guides-20261014-esm1b-protocol-clinvar-inframe-indels

areas
dna-genomes; protein-function
contexts
clinical_research
protocol
Scores used directly (no training on these labels); every method scored on the same 3,470 indels. ESM1b scores are pseudo-log-likelihood ratios between mutated and wild-type sequences; baselines score sequence dissimilarity.
version
Supplementary Table 2 sheet 'Table_of_results_indels'
metric
auroc
metric direction
higher
unit
unitless
denominator
3470
limitations
Benchmark built by the ESM1b workflow's authors (author-reported for ESM1b and the baselines).; CADD is the only published predictor compared; the others that score indels are trained on ClinVar.; Per-type ROC-AUCs are printed only for the three ESM1b PLLR scores where available and CADD; the workbook leaves the other cells 'NaN'.
source locator
Results 'ESM1b can predict the effects of multiresidue variants', paragraph beginning 'We compared ESM1b to other models'; Methods 'Generalized effect scores for indels and stop-gain variants' and 'Baseline scores of indel and stop-gain variant effects'; Supplementary Table 2 'Contents'
Related records

Suggest a correction