rewirebio.iobenchmarks
Contested result

FATHMM and Integrated_fitCons: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and Comprehensive assessment of computational algorithms in predicting cancer driver mutations disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks FATHMM above Integrated_fitCons in 6 of 6 comparisons; Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks Integrated_fitCons above FATHMM in 8 of 9 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank FATHMM above Integrated_fitCons in 7 of their 16 comparisons and Integrated_fitCons higher in 8; 1 is a tie. Of those papers, Comprehensive assessment… ranks Integrated_fitCons higher on its own (8 of 10).

Papers grouped by which system they rank higher: FATHMM or Integrated_fitConsRank FATHMM higher: Clinical likelihood ratios and balanced…. Rank Integrated_fitCons higher: Comprehensive assessment…. Mixed: none.Rank FATHMM higherMixedRank Integrated_fitCons higherClinical likelihoodratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genesComprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations
  • Rank FATHMM higher (1): Clinical likelihood ratios and balanced…
  • Rank Integrated_fitCons higher (1): Comprehensive assessment…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores FATHMM or Integrated_fitCons higherComprehensive assessment…: FATHMM higher in 1, Integrated_fitCons higher in 8, 1 tie. Clinical likelihood ratios and balanced…: FATHMM higher in 6, Integrated_fitCons higher in 0, 0 ties.Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations1 FATHMM higher11 tie18 Integrated_fitCons higher8Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes6 FATHMM higher6
FATHMM higherTieIntegrated_fitCons higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Comprehensive assessment… (10)
ProtocolDatasetMetricFATHMMIntegrated_fitConsHigher
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Accuracy (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.50.51Integrated_fitCons
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Negative predictive value (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.50.51Integrated_fitCons
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Precision (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.50.51Integrated_fitCons
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Recall (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.50.56Integrated_fitCons
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Specificity (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.50.46FATHMM
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Accuracy (cell viability drivers versus neutral, median-score threshold), fraction0.460.49Integrated_fitCons
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Negative predictive value (cell viability drivers versus neutral, median-score threshold), fraction0.460.49Integrated_fitCons
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Precision (cell viability drivers versus neutral, median-score threshold), fraction0.460.49Integrated_fitCons
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Recall (cell viability drivers versus neutral, median-score threshold), fraction0.460.53Integrated_fitCons
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Specificity (cell viability drivers versus neutral, median-score threshold), fraction0.460.46Tie
Comparisons in Clinical likelihood ratios and balanced… (6)
ProtocolDatasetMetricFATHMMIntegrated_fitConsHigher
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingBenignity likelihood ratio (Source 'negative likelihood ratio' (TNR/FNR): likelihood ratio for benignity of a tolerated call, BRCA1 functional truth set), unitless2.41.34FATHMM
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingFalse negatives (Functionally deleterious BRCA1 variants called tolerated), count4064FATHMM
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingLikelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA1 functional truth set), unitless1.21.07FATHMM
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairBenignity likelihood ratio (Source 'negative likelihood ratio' (TNR/FNR): likelihood ratio for benignity of a tolerated call, BRCA2 functional truth set), unitless6.130.91FATHMM
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairFalse negatives (Functionally deleterious BRCA2 variants called tolerated), count556FATHMM
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairLikelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA2 functional truth set), unitless1.770.615FATHMM

What disagrees

studies
2
Compared
FATHMM, Integrated_fitCons
Study that disagrees
Other studies
Rule
two-study-conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-76d25ae738f80d2a · All contested results · How findings are made and followed up