Contested result

CADD and CHASM: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks CADD above CHASM in 2 of 2 comparisons; other studies rank CHASM above CADD in 9 of 9.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Comprehensive assessment… ranks CHASM above CADD in all 9 comparisons that are not ties (1 is a tie). Clinical likelihood ratios and balanced… ranks CADD above CHASM in all 2 comparisons that are not ties (2 are ties).

Rank CHASM higher (1): Comprehensive assessment…
Rank CADD higher (1): Clinical likelihood ratios and balanced… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

CHASM higherTieCADD higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks CHASM higher	Ranks CADD higher	Ties
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and 1 more record · DOI	Disagrees	0	2	2
Comprehensive assessment of computational algorithms in predicting cancer driver mutations and 4 more records · DOI	Agrees	9	0	1

Comparisons in Clinical likelihood ratios and balanced… (4)

Protocol	Dataset	Metric	CHASM	CADD	Higher
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)	BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editing	False negatives (Functionally deleterious BRCA1 variants called tolerated), count	0	0	Tie
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)	BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editing	Likelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA1 functional truth set), unitless	1	1.1	CADD
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)	BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repair	False negatives (Functionally deleterious BRCA2 variants called tolerated), count	0	0	Tie
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)	BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repair	Likelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA2 functional truth set), unitless	1	1.07	CADD

Comparisons in Comprehensive assessment… (10)

Protocol	Dataset	Metric	CHASM	CADD	Higher
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)	Accuracy (OncoKB benchmark (positive set not stated), median-score threshold), fraction	0.65	0.64	CHASM
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)	Negative predictive value (OncoKB benchmark (positive set not stated), median-score threshold), fraction	0.65	0.64	CHASM
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)	Precision (OncoKB benchmark (positive set not stated), median-score threshold), fraction	0.65	0.64	CHASM
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)	Recall (OncoKB benchmark (positive set not stated), median-score threshold), fraction	0.65	0.64	CHASM
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)	Specificity (OncoKB benchmark (positive set not stated), median-score threshold), fraction	0.64	0.64	Tie
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)	Accuracy (cell viability drivers versus neutral, median-score threshold), fraction	0.69	0.59	CHASM
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)	Negative predictive value (cell viability drivers versus neutral, median-score threshold), fraction	0.69	0.59	CHASM
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)	Precision (cell viability drivers versus neutral, median-score threshold), fraction	0.69	0.59	CHASM
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)	Recall (cell viability drivers versus neutral, median-score threshold), fraction	0.69	0.59	CHASM
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)	Specificity (cell viability drivers versus neutral, median-score threshold), fraction	0.69	0.59	CHASM

What disagrees

Other studies: rank CHASM above CADD in 9 of 9 comparisons
This study: ranks CADD above CHASM in 2 of 2 comparisons

Compared: CADD, CHASM
Study that disagrees: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes, Cubuk et al. 2021, Supplementary tables 1-13 (41436_2021_1265_MOESM3_ESM.xlsx)
Other studies: Comprehensive assessment of computational algorithms in predicting cancer driver mutations, Chen et al. 2020, Additional file 10: Performance metrics of 17 algorithms, default categories, benchmark 2 (OncoKB), Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability), Chen et al. 2020, Additional file 22: Performance metrics of 17 algorithms, default categories, benchmark 5 (cell viability), Chen et al. 2020, Additional file 9: Performance metrics of 33 algorithms, median-score threshold, benchmark 2 (OncoKB)
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (4)
Study	Tool	Protocol or benchmark	Metric	Value
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes; Cubuk et al. 2021, Supplementary tables 1-13 (41436_2021_1265_MOESM3_ESM.xlsx)	CADD_phred_a (Cubuk et al. 2021)	BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)	Likelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA1 functional truth set)	1.1
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes; Cubuk et al. 2021, Supplementary tables 1-13 (41436_2021_1265_MOESM3_ESM.xlsx)	CADD_phred_a (Cubuk et al. 2021)	BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)	Likelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA2 functional truth set)	1.07
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes; Cubuk et al. 2021, Supplementary tables 1-13 (41436_2021_1265_MOESM3_ESM.xlsx)	CHASM (Cubuk et al. 2021)	BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)	Likelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA1 functional truth set)	1
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes; Cubuk et al. 2021, Supplementary tables 1-13 (41436_2021_1265_MOESM3_ESM.xlsx)	CHASM (Cubuk et al. 2021)	BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)	Likelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA2 functional truth set)	1

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-8c46ff407b63a77e · All contested results · How findings are made and followed up

Study	Tool	Protocol or benchmark	Metric	Value
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 9: Performance metrics of 33 algorithms, median-score threshold, benchmark 2 (OncoKB)	CADD (Chen et al. 2020)	OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	Accuracy (OncoKB benchmark (positive set not stated), median-score threshold)	0.64
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 9: Performance metrics of 33 algorithms, median-score threshold, benchmark 2 (OncoKB)	CADD (Chen et al. 2020)	OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	Negative predictive value (OncoKB benchmark (positive set not stated), median-score threshold)	0.64
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 9: Performance metrics of 33 algorithms, median-score threshold, benchmark 2 (OncoKB)	CADD (Chen et al. 2020)	OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	Precision (OncoKB benchmark (positive set not stated), median-score threshold)	0.64
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 9: Performance metrics of 33 algorithms, median-score threshold, benchmark 2 (OncoKB)	CADD (Chen et al. 2020)	OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	Recall (OncoKB benchmark (positive set not stated), median-score threshold)	0.64
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability)	CADD (Chen et al. 2020)	Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Accuracy (cell viability drivers versus neutral, median-score threshold)	0.59
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability)	CADD (Chen et al. 2020)	Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Negative predictive value (cell viability drivers versus neutral, median-score threshold)	0.59
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability)	CADD (Chen et al. 2020)	Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Precision (cell viability drivers versus neutral, median-score threshold)	0.59
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability)	CADD (Chen et al. 2020)	Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Recall (cell viability drivers versus neutral, median-score threshold)	0.59
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability)	CADD (Chen et al. 2020)	Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Specificity (cell viability drivers versus neutral, median-score threshold)	0.59
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 9: Performance metrics of 33 algorithms, median-score threshold, benchmark 2 (OncoKB)	CHASM (Chen et al. 2020)	OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	Accuracy (OncoKB benchmark (positive set not stated), median-score threshold)	0.65
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 9: Performance metrics of 33 algorithms, median-score threshold, benchmark 2 (OncoKB)	CHASM (Chen et al. 2020)	OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	Negative predictive value (OncoKB benchmark (positive set not stated), median-score threshold)	0.65
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 9: Performance metrics of 33 algorithms, median-score threshold, benchmark 2 (OncoKB)	CHASM (Chen et al. 2020)	OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	Precision (OncoKB benchmark (positive set not stated), median-score threshold)	0.65
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 9: Performance metrics of 33 algorithms, median-score threshold, benchmark 2 (OncoKB)	CHASM (Chen et al. 2020)	OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)	Recall (OncoKB benchmark (positive set not stated), median-score threshold)	0.65
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability)	CHASM (Chen et al. 2020)	Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Accuracy (cell viability drivers versus neutral, median-score threshold)	0.69
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability)	CHASM (Chen et al. 2020)	Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Negative predictive value (cell viability drivers versus neutral, median-score threshold)	0.69
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability)	CHASM (Chen et al. 2020)	Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Precision (cell viability drivers versus neutral, median-score threshold)	0.69
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability)	CHASM (Chen et al. 2020)	Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Recall (cell viability drivers versus neutral, median-score threshold)	0.69
Comprehensive assessment of computational algorithms in predicting cancer driver mutations; Chen et al. 2020, Additional file 21: Performance metrics of 33 algorithms, median-score threshold, benchmark 5 (cell viability)	CHASM (Chen et al. 2020)	Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)	Specificity (cell viability drivers versus neutral, median-score threshold)	0.69