rewirebio.iobenchmarks
Contested result

MutationAssessor and PROVEAN: An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks PROVEAN above MutationAssessor in 6 of 6 comparisons; 2 of 2 other studies rank MutationAssessor above PROVEAN.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

5 papers rank MutationAssessor above PROVEAN in 21 of their 38 comparisons and PROVEAN higher in 15; 2 are ties. Of those papers, ProteinGym ranks PROVEAN higher on its own (1 of 1). An openly licensed benchmark… ranks PROVEAN above MutationAssessor in all 6 of its comparisons.

Papers grouped by which system they rank higher: MutationAssessor or PROVEANRank MutationAssessor higher: Livesey and Marsh 2025, Clinical likelihood ratios and balanced…, Brandes et al. 2023. Rank PROVEAN higher: An openly licensed benchmark… (disagrees), ProteinGym. Mixed: Comprehensive assessment….Rank MutationAssessor higherMixedRank PROVEAN higherLivesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)Clinical likelihoodratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genesBrandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methodsComprehensiveassessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutationsAn openly licensedbenchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer driversProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design
  • Rank MutationAssessor higher (3): Livesey and Marsh 2025, Clinical likelihood ratios and balanced…, Brandes et al. 2023
  • Rank PROVEAN higher (2): An openly licensed benchmark… (disagrees), ProteinGym
  • Mixed (1): Comprehensive assessment…

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores MutationAssessor or PROVEAN higherAn openly licensed benchmark… (disagrees): MutationAssessor higher in 0, PROVEAN higher in 6, 0 ties. Comprehensive assessment…: MutationAssessor higher in 8, PROVEAN higher in 12, 0 ties. Livesey and Marsh 2025: MutationAssessor higher in 7, PROVEAN higher in 0, 0 ties. Clinical likelihood ratios and balanced…: MutationAssessor higher in 4, PROVEAN higher in 2, 0 ties. Brandes et al. 2023: MutationAssessor higher in 2, PROVEAN higher in 0, 2 ties. ProteinGym: MutationAssessor higher in 0, PROVEAN higher in 1, 0 ties.Disagrees: An openly licensed benchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers6 PROVEAN higher6Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations8 MutationAssessor higher812 PROVEAN higher12Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MutationAssessor higher7Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes4 MutationAssessor higher42 PROVEAN higher2Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods2 MutationAssessor higher22 tie2ProteinGymProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design1 PROVEAN higher1
MutationAssessor higherTiePROVEAN higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in An openly licensed benchmark… (6)
ProtocolDatasetMetricMutationAssessorPROVEANHigher
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, all genes), unitless0.7940.819PROVEAN
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, all genes), unitless0.5540.588PROVEAN
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, oncogenes (gain of function)), unitless0.7690.813PROVEAN
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, oncogenes (gain of function)), unitless0.6040.646PROVEAN
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAUROC (CGC missense, tumour suppressors (loss of function)), unitless0.8240.859PROVEAN
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)Missense variants in 768 Cancer Gene Census genes with open oncogenicity labelsAverage precision (CGC missense, tumour suppressors (loss of function)), unitless0.5350.548PROVEAN
Comparisons in Comprehensive assessment… (20)
ProtocolDatasetMetricMutationAssessorPROVEANHigher
OncoKB benchmark (positive set not stated), default categorical calls (Chen et al. 2020 Additional file 10)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Accuracy (OncoKB benchmark (positive set not stated), default categorical calls), fraction0.580.67PROVEAN
OncoKB benchmark (positive set not stated), default categorical calls (Chen et al. 2020 Additional file 10)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Negative predictive value (OncoKB benchmark (positive set not stated), default categorical calls), fraction0.590.8PROVEAN
OncoKB benchmark (positive set not stated), default categorical calls (Chen et al. 2020 Additional file 10)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Precision (OncoKB benchmark (positive set not stated), default categorical calls), fraction0.570.61PROVEAN
OncoKB benchmark (positive set not stated), default categorical calls (Chen et al. 2020 Additional file 10)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Recall (OncoKB benchmark (positive set not stated), default categorical calls), fraction0.660.89PROVEAN
OncoKB benchmark (positive set not stated), default categorical calls (Chen et al. 2020 Additional file 10)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Specificity (OncoKB benchmark (positive set not stated), default categorical calls), fraction0.50.45MutationAssessor
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Accuracy (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.590.69PROVEAN
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Negative predictive value (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.590.7PROVEAN
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Precision (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.590.69PROVEAN
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Recall (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.610.7PROVEAN
OncoKB benchmark (positive set not stated), median-score threshold (Chen et al. 2020 Additional file 9)OncoKB-annotated somatic missense mutations (oncogenic, likely oncogenic and likely neutral)Specificity (OncoKB benchmark (positive set not stated), median-score threshold), fraction0.570.68PROVEAN
Cell viability drivers versus neutral, default categorical calls (Chen et al. 2020 Additional file 22)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Accuracy (cell viability drivers versus neutral, default categorical calls), fraction0.60.62PROVEAN
Cell viability drivers versus neutral, default categorical calls (Chen et al. 2020 Additional file 22)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Negative predictive value (cell viability drivers versus neutral, default categorical calls), fraction0.590.7PROVEAN
Cell viability drivers versus neutral, default categorical calls (Chen et al. 2020 Additional file 22)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Precision (cell viability drivers versus neutral, default categorical calls), fraction0.60.57MutationAssessor
Cell viability drivers versus neutral, default categorical calls (Chen et al. 2020 Additional file 22)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Recall (cell viability drivers versus neutral, default categorical calls), fraction0.60.78PROVEAN
Cell viability drivers versus neutral, default categorical calls (Chen et al. 2020 Additional file 22)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Specificity (cell viability drivers versus neutral, default categorical calls), fraction0.60.48MutationAssessor
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Accuracy (cell viability drivers versus neutral, median-score threshold), fraction0.60.59MutationAssessor
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Negative predictive value (cell viability drivers versus neutral, median-score threshold), fraction0.60.59MutationAssessor
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Precision (cell viability drivers versus neutral, median-score threshold), fraction0.60.58MutationAssessor
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Recall (cell viability drivers versus neutral, median-score threshold), fraction0.60.59MutationAssessor
Cell viability drivers versus neutral, median-score threshold (Chen et al. 2020 Additional file 21)Missense mutations with Ba/F3 and MCF10A cell viability calls (published and new)Specificity (cell viability drivers versus neutral, median-score threshold), fraction0.60.58MutationAssessor
Comparisons in Livesey and Marsh 2025 (7)
ProtocolDatasetMetricMutationAssessorPROVEANHigher
Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missenseLivesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent51.448.6MutationAssessor
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent58.848.1MutationAssessor
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent10.35.47MutationAssessor
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent59.351MutationAssessor
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent58.346MutationAssessor
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions onlyLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent58.848.3MutationAssessor
Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filledLivesey and Marsh 2025 human DMS benchmark set (36 proteins)Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent57.350MutationAssessor
Comparisons in Clinical likelihood ratios and balanced… (6)
ProtocolDatasetMetricMutationAssessorPROVEANHigher
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingBenignity likelihood ratio (Source 'negative likelihood ratio' (TNR/FNR): likelihood ratio for benignity of a tolerated call, BRCA1 functional truth set), unitless6.641MutationAssessor
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingFalse negatives (Functionally deleterious BRCA1 variants called tolerated), count30366MutationAssessor
BRCA1 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA1 functional truth set: 1641 missense variants classed deleterious or tolerated by HAP1 cell survival after saturation genome editingLikelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA1 functional truth set), unitless2.16.8PROVEAN
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairBenignity likelihood ratio (Source 'negative likelihood ratio' (TNR/FNR): likelihood ratio for benignity of a tolerated call, BRCA2 functional truth set), unitless9.881.17MutationAssessor
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairFalse negatives (Functionally deleterious BRCA2 variants called tolerated), count152MutationAssessor
BRCA2 functional truth set, binary in silico calls (Cubuk et al. 2021)BRCA2 functional truth set: 188 missense variants classed deleterious or tolerated by Homology-directed repairLikelihood ratio (Positive likelihood ratio of a deleterious call (TPR/FPR), BRCA2 functional truth set), unitless1.163.84PROVEAN
Comparisons in Brandes et al. 2023 (4)
Comparisons in ProteinGym (1)
ProtocolDatasetMetricMutationAssessorPROVEANHigher
ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)ProteinGym clinical substitution benchmark at commit 144fe22AUROC (ProteinGym clinical substitutions, zero-shot, Average_AUC over proteins), unitless0.8770.886PROVEAN

What disagrees

Other studies
rank MutationAssessor above PROVEAN in 9 of 9 comparisons
This study
ranks PROVEAN above MutationAssessor in 6 of 6 comparisons
Compared
MutationAssessor, PROVEAN
Study that disagrees
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers, OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)
Other studies
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods, Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML), Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8)
Rule
Ordering reversal; unknown
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (12)
StudyToolProtocol or benchmarkMetricValue
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)MutationAssessor rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)Average precision (CGC missense, all genes)0.554
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)MutationAssessor rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)AUROC (CGC missense, all genes)0.794
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)MutationAssessor rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)Average precision (CGC missense, oncogenes (gain of function))0.604
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)MutationAssessor rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)AUROC (CGC missense, oncogenes (gain of function))0.769
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)MutationAssessor rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)Average precision (CGC missense, tumour suppressors (loss of function))0.535
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)MutationAssessor rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)AUROC (CGC missense, tumour suppressors (loss of function))0.824
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)PROVEAN rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)Average precision (CGC missense, all genes)0.588
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)PROVEAN rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)AUROC (CGC missense, all genes)0.819
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)PROVEAN rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)Average precision (CGC missense, oncogenes (gain of function))0.646
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)PROVEAN rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)AUROC (CGC missense, oncogenes (gain of function))0.813
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)PROVEAN rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)Average precision (CGC missense, tumour suppressors (loss of function))0.548
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)PROVEAN rank score, dbNSFP 5.3.1a (Lee 2026)CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)AUROC (CGC missense, tumour suppressors (loss of function))0.859
In the other studies (18)
StudyToolProtocol or benchmarkMetricValue
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)MutationAssessor (Brandes et al. 2023 clinical benchmarks)ClinVar missense pathogenicity, variants scored by all 19 eligible methods (Brandes et al. 2023 Fig. 2e-f, Supplementary Table 2)AUPRC (balanced PRC-AUC: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot)0.857
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)PROVEAN (Brandes et al. 2023 clinical benchmarks)ClinVar missense pathogenicity, variants scored by all 19 eligible methods (Brandes et al. 2023 Fig. 2e-f, Supplementary Table 2)AUPRC (balanced PRC-AUC: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot)0.844
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)MutationAssessor (Brandes et al. 2023 clinical benchmarks)HGMD/gnomAD missense pathogenicity, variants scored by all 19 eligible methods (Brandes et al. 2023 Fig. 2e-f, Supplementary Table 2)AUPRC (balanced PRC-AUC: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot)0.876
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)PROVEAN (Brandes et al. 2023 clinical benchmarks)HGMD/gnomAD missense pathogenicity, variants scored by all 19 eligible methods (Brandes et al. 2023 Fig. 2e-f, Supplementary Table 2)AUPRC (balanced PRC-AUC: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot)0.864
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)MutationAssessor (Livesey and Marsh 2025)Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missensePairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each)51.4
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)PROVEAN (Livesey and Marsh 2025)Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missensePairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each)48.6
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)MutationAssessor (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsPairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets)58.8
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)MutationAssessor (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsPairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets)10.3
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)PROVEAN (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsPairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets)48.1
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)PROVEAN (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS setsPairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets)5.47
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)MutationAssessor (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays onlyPairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only)59.3
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)PROVEAN (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays onlyPairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only)51
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)MutationAssessor (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays onlyPairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only)58.3
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)PROVEAN (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays onlyPairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only)46
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)MutationAssessor (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filledPairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score)57.3
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)PROVEAN (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filledPairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score)50
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)MutationAssessor (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions onlyPairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only)58.8
Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)PROVEAN (Livesey and Marsh 2025)Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions onlyPairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only)48.3

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-adecfed30105a5a9 · All contested results · How findings are made and followed up