Contested result

MutationTaster and ESM-1b: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

Low severity
Not yet reviewed
two-study-conflict
unknown

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks ESM-1b above MutationTaster in 4 of 4 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks MutationTaster above ESM-1b in 5 of 5 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank MutationTaster above ESM-1b in 5 of their 10 comparisons and ESM-1b higher in 4; 1 is a tie. Of those papers, Brandes et al. 2023 ranks ESM-1b higher on its own (4 of 4).

Rank MutationTaster higher (1): An openly licensed benchmark…
Rank ESM-1b higher (1): Brandes et al. 2023
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

MutationTaster higherTieESM-1b higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks MutationTaster higher	Ranks ESM-1b higher	Ties
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers and 1 more record · DOI	Agrees	5	0	1
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · DOI	Agrees	0	4	0

Comparisons in An openly licensed benchmark… (6)

Protocol	Dataset	Metric	MutationTaster	ESM-1b	Higher
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, all genes), unitless	0.847	0.842	Tie
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, all genes), unitless	0.676	0.641	MutationTaster
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, oncogenes (gain of function)), unitless	0.868	0.841	MutationTaster
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, oncogenes (gain of function)), unitless	0.779	0.719	MutationTaster
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, tumour suppressors (loss of function)), unitless	0.897	0.858	MutationTaster
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, tumour suppressors (loss of function)), unitless	0.616	0.594	MutationTaster

Comparisons in Brandes et al. 2023 (4)

Protocol	Dataset	Metric	MutationTaster	ESM-1b	Higher
ClinVar missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	ClinVar missense variants scored by both ESM1b and MutationTaster (Brandes et al. 2023)	AUPRC (balanced PRC-AUC over the variants both methods score: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot), unitless	0.847	0.906	ESM-1b
ClinVar missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	ClinVar missense variants scored by both ESM1b and MutationTaster (Brandes et al. 2023)	AUROC (global ROC-AUC over the variants both methods score; zero-shot), unitless	0.833	0.906	ESM-1b
HGMD/gnomAD missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	HGMD/gnomAD missense variants scored by both ESM1b and MutationTaster (Brandes et al. 2023)	AUPRC (balanced PRC-AUC over the variants both methods score: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot), unitless	0.884	0.906	ESM-1b
HGMD/gnomAD missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	HGMD/gnomAD missense variants scored by both ESM1b and MutationTaster (Brandes et al. 2023)	AUROC (global ROC-AUC over the variants both methods score; zero-shot), unitless	0.88	0.894	ESM-1b

What disagrees

studies: 2

Compared: MutationTaster, ESM-1b
Study that disagrees
Other studies
Rule: two-study-conflict; unknown
Model identity: Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-96f80fe3048b5630 · All contested results · How findings are made and followed up