Contested result

MutFormer and REVEL: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

Low severity
Not yet reviewed
two-study-conflict
unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above REVEL in 6 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks REVEL above MutFormer in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank MutFormer above REVEL in 6 of their 13 comparisons and REVEL higher in 7. Of those papers, An openly licensed benchmark… ranks REVEL higher on its own (6 of 6).

Rank MutFormer higher (1): Livesey and Marsh 2025
Rank REVEL higher (1): An openly licensed benchmark…
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

MutFormer higherTieREVEL higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks MutFormer higher	Ranks REVEL higher	Ties
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and 1 more record · DOI	Agrees	6	1	0
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers and 1 more record · DOI	Agrees	0	6	0

Comparisons in Livesey and Marsh 2025 (7)

Protocol	Dataset	Metric	MutFormer	REVEL	Higher
Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense	Livesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent	64	76	REVEL
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets), percent	71.4	66.1	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets), percent	17.2	13.2	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only), percent	74.7	65.6	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only), percent	68.9	66.3	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only), percent	71.4	66.1	MutFormer
Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent	70.5	67.5	MutFormer

Comparisons in An openly licensed benchmark… (6)

Protocol	Dataset	Metric	MutFormer	REVEL	Higher
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, all genes), unitless	0.693	0.816	REVEL
CGC missense oncogenicity, all genes (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, all genes), unitless	0.446	0.652	REVEL
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, oncogenes (gain of function)), unitless	0.768	0.843	REVEL
CGC missense oncogenicity, oncogenes (gain of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, oncogenes (gain of function)), unitless	0.582	0.739	REVEL
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	AUROC (CGC missense, tumour suppressors (loss of function)), unitless	0.655	0.859	REVEL
CGC missense oncogenicity, tumour suppressors (loss of function) (Lee 2026, OncoCal tool_performance.tsv)	Missense variants in 768 Cancer Gene Census genes with open oncogenicity labels	Average precision (CGC missense, tumour suppressors (loss of function)), unitless	0.286	0.618	REVEL

What disagrees

studies: 2

Compared: MutFormer, REVEL
Study that disagrees
Other studies
Rule: two-study-conflict; unknown
Model identity: Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-5dd687ad39b2ab94 · All contested results · How findings are made and followed up