rewirebio.iobenchmarks
Contested result

MetaLR and PolyPhen-2: Comprehensive assessment of computational algorithms in predicting cancer driver mutations and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks PolyPhen-2 above MetaLR in 17 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above PolyPhen-2 in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

4 papers rank MetaLR above PolyPhen-2 in 13 of their 38 comparisons and PolyPhen-2 higher in 22; 3 are ties. Of those papers, Comprehensive assessment… ranks PolyPhen-2 higher on its own (17 of 20).

Papers grouped by which system they rank higher: MetaLR or PolyPhen-2Rank MetaLR higher: Livesey and Marsh 2025. Rank PolyPhen-2 higher: Comprehensive assessment…. Mixed: Clinical likelihood ratios and balanced…, An openly licensed benchmark….Rank MetaLR higherMixedRank PolyPhen-2 higherLivesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)Clinical likelihoodratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genesAn openly licensedbenchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer driversComprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations
  • Rank MetaLR higher (1): Livesey and Marsh 2025
  • Rank PolyPhen-2 higher (1): Comprehensive assessment…
  • Mixed (2): Clinical likelihood ratios and balanced…, An openly licensed benchmark…

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores MetaLR or PolyPhen-2 higherComprehensive assessment…: MetaLR higher in 2, PolyPhen-2 higher in 17, 1 tie. Livesey and Marsh 2025: MetaLR higher in 6, PolyPhen-2 higher in 0, 0 ties. Clinical likelihood ratios and balanced…: MetaLR higher in 3, PolyPhen-2 higher in 2, 1 tie. An openly licensed benchmark…: MetaLR higher in 2, PolyPhen-2 higher in 3, 1 tie.Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations2 MetaLR higher21 tie17 PolyPhen-2 higher17Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)6 MetaLR higher6Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes3 MetaLR higher31 tie12 PolyPhen-2 higher2An openly licensed benchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers2 MetaLR higher21 tie13 PolyPhen-2 higher3
MetaLR higherTiePolyPhen-2 higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Comprehensive assessment… (20)
ProtocolDatasetMetricMetaLRPolyPhen-2Higher
OncoKB benchmark (positive set not stated), default categorical calls (Chen et al. 2020 Additional file 10)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Accuracy (OncoKB benchmark (positive set not stated), default categorical calls), fraction0.570.62PolyPhen-2
OncoKB benchmark (positive set not stated), default categorical calls (Chen et al. 2020 Additional file 10)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Negative predictive value (OncoKB benchmark (positive set not stated), default categorical calls), fraction0.590.79PolyPhen-2
OncoKB benchmark (positive set not stated), default categorical calls (Chen et al. 2020 Additional file 10)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Precision (OncoKB benchmark (positive set not stated), default categorical calls), fraction0.560.58PolyPhen-2
OncoKB benchmark (positive set not stated), default categorical calls (Chen et al. 2020 Additional file 10)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Recall (OncoKB benchmark (positive set not stated), default categorical calls), fraction0.670.93PolyPhen-2
OncoKB benchmark (positive set not stated), default categorical calls (Chen et al. 2020 Additional file 10)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Specificity (OncoKB benchmark (positive set not stated), default categorical calls), fraction0.470.29MetaLR
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Accuracy (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.570.63PolyPhen-2
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Negative predictive value (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.570.63PolyPhen-2
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Precision (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.570.62PolyPhen-2
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Recall (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.570.64PolyPhen-2
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Specificity (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.570.62PolyPhen-2
Cell viability drivers versus neutral, default categorical calls (Chen et al. 2020 Additional file 22)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Accuracy (cell viability drivers versus neutral, default categorical calls), fraction0.560.59PolyPhen-2
Cell viability drivers versus neutral, default categorical calls (Chen et al. 2020 Additional file 22)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Negative predictive value (cell viability drivers versus neutral, default categorical calls), fraction0.550.7PolyPhen-2
Cell viability drivers versus neutral, default categorical calls (Chen et al. 2020 Additional file 22)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Precision (cell viability drivers versus neutral, default categorical calls), fraction0.560.56Tie
Cell viability drivers versus neutral, default categorical calls (Chen et al. 2020 Additional file 22)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Recall (cell viability drivers versus neutral, default categorical calls), fraction0.530.88PolyPhen-2
Cell viability drivers versus neutral, default categorical calls (Chen et al. 2020 Additional file 22)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Specificity (cell viability drivers versus neutral, default categorical calls), fraction0.590.29MetaLR
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Accuracy (cell viability drivers versus neutral, median-score threshold), fraction0.560.58PolyPhen-2
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Negative predictive value (cell viability drivers versus neutral, median-score threshold), fraction0.560.58PolyPhen-2
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Precision (cell viability drivers versus neutral, median-score threshold), fraction0.560.58PolyPhen-2
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Recall (cell viability drivers versus neutral, median-score threshold), fraction0.560.58PolyPhen-2
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Specificity (cell viability drivers versus neutral, median-score threshold), fraction0.560.57PolyPhen-2
Comparisons in Livesey and Marsh 2025 (6)
ProtocolDatasetMetricMetaLRPolyPhen-2Higher
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent45.742.4MetaLR
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent8.136.76MetaLR
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent42.141.5MetaLR
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent48.343.1MetaLR
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent45.742.4MetaLR
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filledLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent46.644.1MetaLR
Comparisons in Clinical likelihood ratios and balanced… (6)
ProtocolDatasetMetricMetaLRPolyPhen-2Higher
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingBenignity likelihood ratio (Source 'negative likelihood ratio' (TNR/FNR): likelihood ratio for benignity of a tolerated call, BRCA1 functional truth set), unitless6.0527.6PolyPhen-2
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingFalse negatives (Functionally deleterious BRCA1 variants called tolerated), count182PolyPhen-2
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingLikelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA1 functional truth set), unitless1.341.16MetaLR
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairBenignity likelihood ratio (Source 'negative likelihood ratio' (TNR/FNR): likelihood ratio for benignity of a tolerated call, BRCA2 functional truth set), unitless21.87.28MetaLR
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairFalse negatives (Functionally deleterious BRCA2 variants called tolerated), count11Tie
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairLikelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA2 functional truth set), unitless1.491.11MetaLR
Comparisons in An openly licensed benchmark… (6)
ProtocolDatasetMetricMetaLRPolyPhen-2Higher
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, all genes), unitless0.7480.814PolyPhen-2
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, all genes), unitless0.5440.47MetaLR
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, oncogenes (gain of function)), unitless0.750.805PolyPhen-2
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, oncogenes (gain of function)), unitless0.5430.541Tie
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, tumour suppressors (loss of function)), unitless0.7880.847PolyPhen-2
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, tumour suppressors (loss of function)), unitless0.5220.409MetaLR

What disagrees

studies
2
Compared
MetaLR, PolyPhen-2
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-e69a3e36da1a1ef8 · All contested results · How findings are made and followed up