rewirebio.iobenchmarks
Contested result

VEST4 and BayesDel: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks VEST4 above BayesDel in 6 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks BayesDel above VEST4 in 5 of 5 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

4 papers rank VEST4 above BayesDel in 8 of their 20 comparisons and BayesDel higher in 10; 2 are ties. Of those papers, An openly licensed benchmark…, ProteinGym rank BayesDel higher on their own (5 of 6, 1 of 1).

Papers grouped by which system they rank higher: VEST4 or BayesDelRank VEST4 higher: Livesey and Marsh 2025. Rank BayesDel higher: An openly licensed benchmark…, ProteinGym. Mixed: Clinical likelihood ratios and balanced….Rank VEST4 higherMixedRank BayesDel higherLivesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)Clinical likelihoodratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genesAn openly licensedbenchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer driversProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design
  • Rank VEST4 higher (1): Livesey and Marsh 2025
  • Rank BayesDel higher (2): An openly licensed benchmark…, ProteinGym
  • Mixed (1): Clinical likelihood ratios and balanced…

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores VEST4 or BayesDel higherLivesey and Marsh 2025: VEST4 higher in 6, BayesDel higher in 1, 0 ties. An openly licensed benchmark…: VEST4 higher in 0, BayesDel higher in 5, 1 tie. Clinical likelihood ratios and balanced…: VEST4 higher in 2, BayesDel higher in 3, 1 tie. ProteinGym: VEST4 higher in 0, BayesDel higher in 1, 0 ties.Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)6 VEST4 higher61 BayesDel higher1An openly licensed benchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers1 tie15 BayesDel higher5Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes2 VEST4 higher21 tie13 BayesDel higher3ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design1 BayesDel higher1
VEST4 higherTieBayesDel higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Livesey and Marsh 2025 (7)
ProtocolDatasetMetricVEST4BayesDelHigher
Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missenseLivesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent77.166.9VEST4
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent5956.9VEST4
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent2.335.44BayesDel
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent59.956.8VEST4
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent58.457VEST4
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent5956.9VEST4
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filledLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent60.558.1VEST4
Comparisons in An openly licensed benchmark… (6)
ProtocolDatasetMetricVEST4BayesDelHigher
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, all genes), unitless0.8330.864BayesDel
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, all genes), unitless0.6480.712BayesDel
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, oncogenes (gain of function)), unitless0.840.878BayesDel
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, oncogenes (gain of function)), unitless0.7130.764BayesDel
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, tumour suppressors (loss of function)), unitless0.8850.892Tie
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, tumour suppressors (loss of function)), unitless0.6260.684BayesDel
Comparisons in Clinical likelihood ratios and balanced… (6)
ProtocolDatasetMetricVEST4BayesDelHigher
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingBenignity likelihood ratio (Source 'negative likelihood ratio' (TNR/FNR): likelihood ratio for benignity of a tolerated call, BRCA1 functional truth set), unitless55.211.8VEST4
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingFalse negatives (Functionally deleterious BRCA1 variants called tolerated), count17VEST4
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingLikelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA1 functional truth set), unitless1.171.26BayesDel
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairBenignity likelihood ratio (Source 'negative likelihood ratio' (TNR/FNR): likelihood ratio for benignity of a tolerated call, BRCA2 functional truth set), unitless8.3216.1BayesDel
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairFalse negatives (Functionally deleterious BRCA2 variants called tolerated), count11Tie
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairLikelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA2 functional truth set), unitless1.131.31BayesDel
Comparisons in ProteinGym (1)
ProtocolDatasetMetricVEST4BayesDelHigher
ProteinGym clinical substitutions, supervised predictors, average AUC (commit 144fe22)ProteinGym clinical substitution benchmark at commit 144fe22AUROC (ProteinGym clinical substitutions, supervised clinical predictors, Average_AUC over proteins), unitless0.9290.972BayesDel

What disagrees

studies
2
Compared
VEST4, BayesDel
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-c17b66edca7cc924 · All contested results · How findings are made and followed up