rewirebio.iobenchmarks
← Browse all benchmarks
Benchmark

Livesey and Marsh variant effect predictor benchmark (2025 edition)

Ranks missense variant effect predictors by pairwise comparisons of their Spearman correlation with human deep mutational scanning data, and separately by pairwise AUROC on ClinVar pathogenic versus gnomAD missense variants.

Results are recorded on the linked protocols, tasks or configurations listed below. Their settings and evaluations are kept separate.

6 protocols with results

Overview

An explanatory profile has not yet been reviewed for this record.

Consult the linked sources for architecture or protocol details. Missing evidence is not evidence of a missing capability.

Results on linked records

This benchmark has no results of its own in this release. 6 linked records have results. They are not assigned to the underlying benchmark: each record keeps its own settings and evaluations, and its scores are on its own page.

Methods and evaluation design

Procedure, tasks and evaluated configurations

Baseline coverage

Reference methods help show what a model adds beyond simple controls. We track a null control and a conventional method for each protocol.

0 of 12 active baseline roles have published Rewire measurements in this release. Measurements on a selected protocol do not establish coverage of an entire suite.

Baseline status by linked protocol

Protocol coverage CSV (gzip) · Model evaluation matrix (gzip) · Source table (gzip) · Release and checksums (gzip)

Coverage is derived from release 2026-10-22-1b7cf027e4f6. Source citations describe the original records; they do not validate an unreviewed baseline proposal. No results have been generated by this audit.

Run instructions

No runnable recipe has been reviewed for this benchmark. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.

Strengths, limitations and unresolved questions

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

0 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-22-1b7cf027e4f6
Property and statementOriginal source and locationReview and provenance

No evidence rows match these filters. Choose another scope or clear the search.

How to read the evidence

Sources and history

Release 2026-10-22-1b7cf027e4f6 · Record review: source checked

2 source records and release historyDownload this release (gzip)
Technical metadata and extraction receipts

Stable ID: vep-lit-20261014-benchmark-livesey-marsh-vep

areas
dna-genomes; proteins-complexes
contexts
research; clinical_research
entity level
suite
version
Genome Biol 2025;26:104 (third edition, after Mol Syst Biol 2020 and 2023)
task
Missense variant effect prediction, ranked against functional assays and clinical labels
scope note
97 VEPs on 36 human DMS datasets (one score set per protein, at least 1000 single amino acid substitutions after removing ClinVar and gnomAD variants); 83 VEPs on 819 proteins with at least 10 ClinVar pathogenic or likely pathogenic and 10 gnomAD v4.1 missense variants. Rank score is each VEP's average pairwise win rate against all other VEPs.
source locator
Results 'New DMS datasets and VEPs', 'Comparison of VEP performance using DMS data', 'Performance of VEPs on clinical variant classification'; Methods
limitations
Rank scores are relative: they depend on which other VEPs are in the comparison and are not a per-VEP correlation or AUROC.; The earlier editions (Mol Syst Biol 2020;16:e9380 and 2023;19:e11474) used other DMS sets, VEP sets and, in 2020, another ranking method; they are not extracted here.
review
reviewer: claude; reviewer note: Claude review agent; no human review claimed. Independent Claude (Opus 5.5) review agent, separate from the extraction agent that built this batch. Automated review only. Transcription, not reproduction.; reviewed at: 2026-10-14T15:30:00Z; artifact sha256: 1735ee461ef192d9de67c914e9ad73d72d8aa255ea0e3479eb68cd8631d99fa2; retrieval url: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC12016141/fullTextXML; method: ai-assisted-source-review; scope-review; method note: Descriptive fields, identity links and missing-metadata reasons checked against the cited sources.; note: Checked; see the dated review for rulings on identity and method type.
Related records

Suggest a correction