Use calibrated computational evidence for missense variants (PP3/BP4)

What each column measures:

• Pejaver et al. 2022: four-level intervals, 13 tools: Score interval per ACMG/AMP evidence strength (BP4 and PP3, Supporting to Very Strong) for 13 missense predictors, ClinGen SVI calibration on ClinVar 2019 missense variants
• Bergquist et al. 2025: points scale, 7 tools: Score interval per point value (−4 to +4, including ±3 and indeterminate) for BayesDel, MutPred2, REVEL, VEST4, AlphaMissense, ESM1b and VARITY_R
• Developer thresholds: local likelihood ratio: Local positive likelihood ratio at the developer-recommended thresholds of SIFT, PolyPhen-2 and CADD, against the PP3 Supporting cut-off of 2.406
• Developer thresholds: gnomAD fraction called pathogenic: Fraction of the gnomAD reference set called pathogenic by SIFT, PolyPhen-2 and CADD at their developer thresholds, against a 4.41% prior

• Calibration is an average over genes; Tejura et al. 2024 found that about 70% of the gene-level REVEL and BayesDel intervals with enough control variants exceeded the incorrect predictions the genome-wide recommendations imply (Summary).
• The developers of BayesDel, MutPred2, REVEL and VEST4, the four tools that reach PP3_Strong, and of MPC are authors of Pejaver et al. 2022; those evaluations are recorded as author_reported.
• Table S1 point estimates and confidence bounds were not retrieved; only the printed intervals are recorded.
• Labels are ClinVar classifications, which the authors assume were not driven by computational evidence (Stenton et al. 2024, Table 1).
• Training variants were removed only for BayesDel, FATHMM, MPC, MutPred2, PolyPhen-2, REVEL and VEST4 (and the meta-predictors' constituents) where available; no removal is stated for the other six tools (Methods).
• Calibration variants may not represent the novel variants to be classified (Stenton et al. 2024, Table 1).
• Transcribed from the source; no reproduction by Rewire and no human scientific review.
• The developer thresholds of AlphaMissense (0.564) and ESM1b (−7.5) do not reach Supporting under this calibration (Results).
• EVE could not be calibrated reliably (about half the benign variants unscored) and is not included (Results).
• MutPred2 and REVEL were co-developed by authors of Bergquist et al. 2025 (V. Pejaver, P. Radivojac). The BayesDel, MutPred2, REVEL and VEST4 rows are carried over from Pejaver et al. 2022, whose authors include the developers of all four.
• Some of the previously calibrated tools did not reach the likelihood ratios expected for the 3-point intervals (Results, Fig. 1B).
• The authors state that this calibration shares the previous study's limitations, including potential circularities and gene- or disease-specific variability (Discussion).
• VARITY was trained on ClinVar; each precomputed score comes from a model that excluded that variant (Materials and Methods).
• Evidence strength was judged on the point estimate of the likelihood ratio, not its confidence bound (Table 3 footnote).
• The gnomAD set is unlabelled; a fraction above the prior suggests false positive calls but does not count them.

Protocols: PP3/BP4 score intervals per evidence strength (Pejaver et al. 2022 Table 2), PP3/BP4 score intervals on the points scale (Bergquist et al. 2025 Table 1), Local likelihood ratio at developer-recommended thresholds (Pejaver et al. 2022 Table 3), Share of gnomAD variants called pathogenic at developer thresholds (Pejaver et al. 2022 Table 3)

What each column measures:

• BRCA1 MAVE, Ramadane-Morchadi et al. 2025: Log2 likelihood ratio of BP4 and PP3 score ranges for AlphaMissense, FoldX ΔΔG and BayesDel on 1,519 BRCA1 RING and BRCT missense variants with MAVE labels
• BRCA1 generic BayesDel thresholds (ENIGMA): BRCA1 BP4 coverage and erroneous PP3 assignment when the generic Pejaver BayesDel thresholds are applied to functional reference variants
• BRCA2 generic BayesDel thresholds (ENIGMA): BRCA2 BP4 coverage and erroneous PP3 assignment when the generic Pejaver BayesDel thresholds are applied to functional reference variants

• BRCA1 RING and BRCT domains only; functional labels stand in for pathogenicity.
• Some thresholds were chosen on the same variants (see the protocol's limitations).
• BRCA1 and BRCA2 are excluded from this use case because an expert-panel specification sets their thresholds; this protocol shows how the genome-wide thresholds behave in one well-studied gene, not in genes without a specification.
• Transcribed from the source; no reproduction by Rewire and no human scientific review.
• Functional proxy labels, not independent clinical classifications.
• Primary narrative only; complete Supplementary Table S4 unavailable (protocol limitations).

Protocols: BRCA1 MAVE PP3/BP4 evidence strength by score threshold (Ramadane-Morchadi et al. 2025 Table 1), BRCA1 generic BayesDel threshold evidence-code assignment, BRCA2 generic BayesDel threshold evidence-code assignment

What each column measures:

• DMS benchmark, all 36 DMS sets: Average pairwise win rate of each missense VEP against 96 others, by |Spearman| with 36 human DMS datasets
• DMS benchmark, direct assays only: Average pairwise win rate by |Spearman| with the DMS assays the authors class as direct
• DMS benchmark, indirect assays only: Average pairwise win rate by |Spearman| with the DMS assays the authors class as indirect
• DMS benchmark, SNV-accessible substitutions: Average pairwise win rate by |Spearman| restricted to substitutions reachable by one nucleotide change
• DMS benchmark, SNV-accessible, gaps filled: Average pairwise win rate by |Spearman| on SNV-accessible substitutions with missing predictions set to each VEP's most benign score
• Clinical benchmark, ClinVar P/LP versus gnomAD missense: Average pairwise win rate by per-protein AUROC separating ClinVar P/LP from gnomAD v4.1 missense variants in 819 proteins

• DMS phenotypes are chosen per protein and are not pathogenicity labels.
• Rank scores are relative to the VEP set compared; they are not a correlation and cannot be compared with other tables.
• No calibration to ACMG/AMP evidence strength and no thresholds.
• Correlation is assessed within each protein, so the ranking does not test whether scores are comparable across genes; the authors report that DMS and clinical rankings agree closely for population-free and population-tuned VEPs (Pearson r 0.978 and 0.994) and less for clinical-trained VEPs (Results, Fig. 4).
• Transcribed from the source; no reproduction by Rewire and no human scientific review.
• Which assays are direct is not fully recoverable from Table S1 (four blank assay descriptions).
• Which assays are indirect is not fully recoverable from Table S1.
• The negative class is gnomAD v4.1 missense variants not classed pathogenic ('putatively benign'); the authors note it will include unclassified pathogenic variants, particularly in recessive and incomplete-penetrance genes (Results).
• VEPs that use allele frequency as a feature likely benefit from circularity, because frequency is used to classify variants as benign (Discussion); the gnomAD negative class reduces but does not remove this.
• No training variants were removed; the authors attribute most of the clinical-trained VEPs' advantage over the DMS ranking to circularity (Results, Fig. 4).
• AUROC is computed per protein, so the ranking does not test whether scores are comparable across genes, which genome-wide thresholds assume.

Protocols: Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets, Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays only, Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays only, Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions only, Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filled, Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense

What each column measures:

• Substitutions, zero-shot: Average over proteins of AUC separating pathogenic from benign clinical substitutions, zero-shot models
• Substitutions, supervised predictors: Average over proteins of AUC for supervised clinical predictors from dbNSFP 4.4a

• Only 13 models are listed; the file's completeness is not stated.
• Averaging and uncertainty construction not verified for the clinical summaries.
• Clinical labels are biased towards heavily studied proteins, such as well-known cancer genes, and towards European ancestry (ProteinGym paper, Limitations).
• Evolutionary conservation is one of the criteria used to classify ClinVar variants, so even zero-shot models are not free of circularity (ProteinGym paper, Limitations).
• Per-protein AUC averaged over proteins does not test whether scores are comparable across genes, which genome-wide thresholds assume.
• The ProteinGym paper describes the 2023 clinical sets; whether the sets behind the 144fe22 files are unchanged is not stated.
• Transcribed from the source; no reproduction by Rewire and no human scientific review.
• Do not compare with the zero-shot rows as if on equal footing.
• Performance is reported on all available ClinVar labels, so supervised methods whose training data overlap these labels are likely overestimated; models trained on population frequencies leak further, because most ClinVar benign labels rest on observed frequency (ProteinGym paper, 'Clinical datasets', supervised benchmark).

Protocols: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22), ProteinGym clinical substitutions, supervised predictors, average AUC (commit 144fe22)