Contested result

MutationTaster and ESM-1b: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks ESM-1b above MutationTaster in 4 of 4 comparisons; other studies rank MutationTaster above ESM-1b in 5 of 5.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

An openly licensed benchmark… ranks MutationTaster above ESM-1b in all 5 comparisons that are not ties (1 is a tie). Brandes et al. 2023 ranks ESM-1b above MutationTaster in all 4 of its comparisons.

Rank MutationTaster higher (1): An openly licensed benchmark…
Rank ESM-1b higher (1): Brandes et al. 2023 (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

MutationTaster higherTieESM-1b higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks MutationTaster higher	Ranks ESM-1b higher	Ties
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods · DOI	Disagrees	0	4	0
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers and 1 more record · DOI	Agrees	5	0	1

Comparisons in Brandes et al. 2023 (4)

Protocol	Dataset	Metric	MutationTaster	ESM-1b	Higher
ClinVar missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	ClinVar missense variants scored by both ESM1b and MutationTaster (Brandes et al. 2023)	AUPRC (balanced PRC-AUC over the variants both methods score: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot), unitless	0.847	0.906	ESM-1b
ClinVar missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	ClinVar missense variants scored by both ESM1b and MutationTaster (Brandes et al. 2023)	AUROC (global ROC-AUC over the variants both methods score; zero-shot), unitless	0.833	0.906	ESM-1b
HGMD/gnomAD missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	HGMD/gnomAD missense variants scored by both ESM1b and MutationTaster (Brandes et al. 2023)	AUPRC (balanced PRC-AUC over the variants both methods score: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot), unitless	0.884	0.906	ESM-1b
HGMD/gnomAD missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	HGMD/gnomAD missense variants scored by both ESM1b and MutationTaster (Brandes et al. 2023)	AUROC (global ROC-AUC over the variants both methods score; zero-shot), unitless	0.88	0.894	ESM-1b

Comparisons in An openly licensed benchmark… (6)

Protocol	Dataset	Metric	MutationTaster	ESM-1b	Higher
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, all genes), unitless	0.847	0.842	Tie
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, all genes), unitless	0.676	0.641	MutationTaster
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, oncogenes (gain of function)), unitless	0.868	0.841	MutationTaster
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, oncogenes (gain of function)), unitless	0.779	0.719	MutationTaster
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, tumour suppressors (loss of function)), unitless	0.897	0.858	MutationTaster
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, tumour suppressors (loss of function)), unitless	0.616	0.594	MutationTaster

What disagrees

Other studies: rank MutationTaster above ESM-1b in 5 of 5 comparisons
This study: ranks ESM-1b above MutationTaster in 4 of 4 comparisons

Compared: MutationTaster, ESM-1b
Study that disagrees: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods
Other studies: An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers, OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (8)
Study	Tool	Protocol or benchmark	Metric	Value
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)	ESM1b (Brandes et al. 2023 clinical benchmarks)	ClinVar missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	AUPRC (balanced PRC-AUC over the variants both methods score: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot)	0.906
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)	ESM1b (Brandes et al. 2023 clinical benchmarks)	ClinVar missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	AUROC (global ROC-AUC over the variants both methods score; zero-shot)	0.906
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)	MutationTaster (Brandes et al. 2023 head-to-head comparison)	ClinVar missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	AUPRC (balanced PRC-AUC over the variants both methods score: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot)	0.847
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)	MutationTaster (Brandes et al. 2023 head-to-head comparison)	ClinVar missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	AUROC (global ROC-AUC over the variants both methods score; zero-shot)	0.833
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)	ESM1b (Brandes et al. 2023 clinical benchmarks)	HGMD/gnomAD missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	AUPRC (balanced PRC-AUC over the variants both methods score: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot)	0.906
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)	ESM1b (Brandes et al. 2023 clinical benchmarks)	HGMD/gnomAD missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	AUROC (global ROC-AUC over the variants both methods score; zero-shot)	0.894
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)	MutationTaster (Brandes et al. 2023 head-to-head comparison)	HGMD/gnomAD missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	AUPRC (balanced PRC-AUC over the variants both methods score: mean over 100 random downsamples to equal pathogenic and benign counts (80% of the minority class); zero-shot)	0.884
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods; Genome-wide prediction of disease variant effects with a deep protein language model (Brandes et al., Nature Genetics 2023)	MutationTaster (Brandes et al. 2023 head-to-head comparison)	HGMD/gnomAD missense pathogenicity, ESM1b head to head with one other method on the variants both score (Brandes et al. 2023 Fig. 2g,h, Supplementary Table 2)	AUROC (global ROC-AUC over the variants both methods score; zero-shot)	0.88

In the other studies (10)
Study	Tool	Protocol or benchmark	Metric	Value
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)	ESM1b rank score, dbNSFP 5.3.1a (Lee 2026)	CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)	Average precision (CGC missense, all genes)	0.641
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)	ESM1b rank score, dbNSFP 5.3.1a (Lee 2026)	CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	Average precision (CGC missense, oncogenes (gain of function))	0.719
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)	ESM1b rank score, dbNSFP 5.3.1a (Lee 2026)	CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	AUROC (CGC missense, oncogenes (gain of function))	0.841
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)	ESM1b rank score, dbNSFP 5.3.1a (Lee 2026)	CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	Average precision (CGC missense, tumour suppressors (loss of function))	0.594
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)	ESM1b rank score, dbNSFP 5.3.1a (Lee 2026)	CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	AUROC (CGC missense, tumour suppressors (loss of function))	0.858
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)	MutationTaster rank score, dbNSFP 5.3.1a (Lee 2026)	CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)	Average precision (CGC missense, all genes)	0.676
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)	MutationTaster rank score, dbNSFP 5.3.1a (Lee 2026)	CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	Average precision (CGC missense, oncogenes (gain of function))	0.779
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)	MutationTaster rank score, dbNSFP 5.3.1a (Lee 2026)	CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	AUROC (CGC missense, oncogenes (gain of function))	0.868
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)	MutationTaster rank score, dbNSFP 5.3.1a (Lee 2026)	CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	Average precision (CGC missense, tumour suppressors (loss of function))	0.616
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers; OncoCal repository, tables/tool_performance.tsv (per-tool AUROC and AUPRC by gene role)	MutationTaster rank score, dbNSFP 5.3.1a (Lee 2026)	CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	AUROC (CGC missense, tumour suppressors (loss of function))	0.897

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-c768dcc0ca166eb9 · All contested results · How findings are made and followed up