VEST4 and BayesDel: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree
- Low severity
- Not yet reviewed
- Two-study conflict
- unknown
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks VEST4 above BayesDel in 6 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks BayesDel above VEST4 in 5 of 5 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
4 papers rank VEST4 above BayesDel in 8 of their 20 comparisons and BayesDel higher in 10; 2 are ties. Of those papers, An openly licensed benchmark…, ProteinGym rank BayesDel higher on their own (5 of 6, 1 of 1).
- Rank VEST4 higher (1): Livesey and Marsh 2025
- Rank BayesDel higher (2): An openly licensed benchmark…, ProteinGym
- Mixed (1): Clinical likelihood ratios and balanced…
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks VEST4 higher | Ranks BayesDel higher | Ties |
|---|---|---|---|---|
| Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and 1 more record · DOI | Agrees | 6 | 1 | 0 |
| An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers and 1 more record · DOI | Agrees | 0 | 5 | 1 |
| Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and 1 more record · DOI | Agrees | 2 | 3 | 1 |
| ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and 6 more records · DOI | Agrees | 0 | 1 | 0 |
Comparisons in Livesey and Marsh 2025 (7)
| Protocol | Dataset | Metric | VEST4 | BayesDel | Higher |
|---|---|---|---|---|---|
| Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense | Livesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent | 77.1 | 66.9 | VEST4 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent | 59 | 56.9 | VEST4 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent | 2.33 | 5.44 | BayesDel |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent | 59.9 | 56.8 | VEST4 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent | 58.4 | 57 | VEST4 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent | 59 | 56.9 | VEST4 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filled | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent | 60.5 | 58.1 | VEST4 |
Comparisons in An openly licensed benchmark… (6)
Comparisons in Clinical likelihood ratios and balanced… (6)
Comparisons in ProteinGym (1)
| Protocol | Dataset | Metric | VEST4 | BayesDel | Higher |
|---|---|---|---|---|---|
| ProteinGym clinical substitutions, supervised predictors, average AUC (commit 144fe22) | ProteinGym clinical substitution benchmark at commit 144fe22 | AUROC (ProteinGym clinical substitutions, supervised clinical predictors, Average_AUC over proteins), unitless | 0.929 | 0.972 | BayesDel |
What disagrees
- studies
- 2
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-c17b66edca7cc924 · All contested results · How findings are made and followed up