MutPred and PrimateAI: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

Which studies agree

4 papers rank MutPred above PrimateAI in 16 of their 20 comparisons and PrimateAI higher in 4. Of those papers, Clinical likelihood ratios and balanced… ranks PrimateAI higher on its own (4 of 6). Comprehensive assessment… ranks PrimateAI above MutPred in 9 of its 10 comparisons and MutPred higher in 1.

What disagrees

Other studies

rank MutPred above PrimateAI in 16 of 20 comparisons

This study

ranks PrimateAI above MutPred in 9 of 10 comparisons

Compared

MutPred, PrimateAI

Study that disagrees

Comprehensive assessment of computational algorithms in predicting cancer driver mutations, Chen et al. 2020, Additional file 10: Performance metrics of 17 algorithms, default categories, benchmark 2 (OncoKB), Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability), Chen et al. 2020, Additional file 22: Performance metrics of 17 algorithms, default categories, benchmark 5 (cell viability), Chen et al. 2020, Additional file 9: Performance metrics of 33 algorithms, median-score threshold, benchmark 2 (OncoKB)

Other studies

11 studies

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes, Cubuk et al. 2021, Supplementary tables 1-13 (41436_2021_1265_MOESM3_ESM.xlsx), An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers, OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role), ProteinGym AMFR assay-level Spearman comparison, Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML), Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8), ProteinGym clinical supervised substitutions AUC summary at commit 144fe22, ProteinGym clinical zero-shot indels AUC summary at commit 144fe22, ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22, ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22

Rule

Ordering reversal; different tasks across studies

Model identity

Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-605fd20ed6a27cf4 · All contested results · How findings are made and followed up