Contested result

MutFormer and ESM-1b: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

Low severity
Not yet reviewed
two-study-conflict
unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above ESM-1b in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks ESM-1b above MutFormer in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank MutFormer above ESM-1b in 7 of their 13 comparisons and ESM-1b higher in 6. Of those papers, An openly licensed benchmark… ranks ESM-1b higher on its own (6 of 6).

Rank MutFormer higher (1): Livesey and Marsh 2025
Rank ESM-1b higher (1): An openly licensed benchmark…
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

MutFormer higherTieESM-1b higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks MutFormer higher	Ranks ESM-1b higher	Ties
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and 1 more record · DOI	Agrees	7	0	0
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers and 1 more record · DOI	Agrees	0	6	0

Comparisons in Livesey and Marsh 2025 (7)

Protocol	Dataset	Metric	MutFormer	ESM-1b	Higher
Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense	Livesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent	64	54.3	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets), percent	71.4	65	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets), percent	17.2	14.7	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only), percent	74.7	62.4	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only), percent	68.9	67	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only), percent	71.4	64.5	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent	70.5	65.9	MutFormer

Comparisons in An openly licensed benchmark… (6)

Protocol	Dataset	Metric	MutFormer	ESM-1b	Higher
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, all genes), unitless	0.693	0.842	ESM-1b
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, all genes), unitless	0.446	0.641	ESM-1b
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, oncogenes (gain of function)), unitless	0.768	0.841	ESM-1b
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, oncogenes (gain of function)), unitless	0.582	0.719	ESM-1b
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, tumour suppressors (loss of function)), unitless	0.655	0.858	ESM-1b
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, tumour suppressors (loss of function)), unitless	0.286	0.594	ESM-1b

What disagrees

studies: 2

Compared: MutFormer, ESM-1b
Study that disagrees
Other studies
Rule: two-study-conflict; unknown
Model identity: Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-6f86ce6958902738 · All contested results · How findings are made and followed up