rewirebio.iobenchmarks
Contested result

DEOGEN2 and MPC: Comprehensive assessment of computational algorithms in predicting cancer driver mutations and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks MPC above DEOGEN2 in 10 of 10 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks DEOGEN2 above MPC in 6 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

4 papers rank DEOGEN2 above MPC in 10 of their 24 comparisons and MPC higher in 13; 1 is a tie. Of those papers, Comprehensive assessment… ranks MPC higher on its own (10 of 10).

Papers grouped by which system they rank higher: DEOGEN2 or MPCRank DEOGEN2 higher: Livesey and Marsh 2025, ProteinGym. Rank MPC higher: Comprehensive assessment…. Mixed: An openly licensed benchmark….Rank DEOGEN2 higherMixedRank MPC higherLivesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and DesignAn openly licensedbenchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer driversComprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations
  • Rank DEOGEN2 higher (2): Livesey and Marsh 2025, ProteinGym
  • Rank MPC higher (1): Comprehensive assessment…
  • Mixed (1): An openly licensed benchmark…

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores DEOGEN2 or MPC higherComprehensive assessment…: DEOGEN2 higher in 0, MPC higher in 10, 0 ties. Livesey and Marsh 2025: DEOGEN2 higher in 6, MPC higher in 1, 0 ties. An openly licensed benchmark…: DEOGEN2 higher in 3, MPC higher in 2, 1 tie. ProteinGym: DEOGEN2 higher in 1, MPC higher in 0, 0 ties.Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations10 MPC higher10Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)6 DEOGEN2 higher61 MPC higher1An openly licensed benchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers3 DEOGEN2 higher31 tie12 MPC higher2ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design1 DEOGEN2 higher1
DEOGEN2 higherTieMPC higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Comprehensive assessment… (10)
ProtocolDatasetMetricDEOGEN2MPCHigher
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Accuracy (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.660.68MPC
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Negative predictive value (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.660.68MPC
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Precision (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.660.68MPC
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Recall (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.660.68MPC
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Specificity (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.660.68MPC
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Accuracy (cell viability drivers versus neutral, median-score threshold), fraction0.640.66MPC
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Negative predictive value (cell viability drivers versus neutral, median-score threshold), fraction0.640.66MPC
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Precision (cell viability drivers versus neutral, median-score threshold), fraction0.640.66MPC
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Recall (cell viability drivers versus neutral, median-score threshold), fraction0.640.66MPC
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Specificity (cell viability drivers versus neutral, median-score threshold), fraction0.640.66MPC
Comparisons in Livesey and Marsh 2025 (7)
ProtocolDatasetMetricDEOGEN2MPCHigher
Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missenseLivesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent60.845DEOGEN2
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent59.453.6DEOGEN2
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent11.411.6MPC
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent66.157.2DEOGEN2
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent54.650.7DEOGEN2
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent59.853.6DEOGEN2
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filledLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent61.154.8DEOGEN2
Comparisons in An openly licensed benchmark… (6)
ProtocolDatasetMetricDEOGEN2MPCHigher
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, all genes), unitless0.8150.823Tie
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, all genes), unitless0.6530.613DEOGEN2
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, oncogenes (gain of function)), unitless0.8580.842DEOGEN2
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, oncogenes (gain of function)), unitless0.7320.716DEOGEN2
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, tumour suppressors (loss of function)), unitless0.8410.873MPC
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, tumour suppressors (loss of function)), unitless0.5960.626MPC
Comparisons in ProteinGym (1)
ProtocolDatasetMetricDEOGEN2MPCHigher
ProteinGym clinical substitutions, supervised predictors, average AUC (commit 144fe22)ProteinGym clinical substitution benchmark at commit 144fe22AUROC (ProteinGym clinical substitutions, supervised clinical predictors, Average_AUC over proteins), unitless0.8940.881DEOGEN2

What disagrees

studies
2
Compared
DEOGEN2, MPC
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-a86e05780cc4b1b7 · All contested results · How findings are made and followed up