rewirebio.iobenchmarks
Evidence and investigation

Contested results

Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.

These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.

A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.

How the check works, and how each follow-up is recorded: discrepancies between studies.

In this release

  • 192open
  • 6settled
  • 7resolved
  • 3medium severity
  • 202low severity

Release 2026-11-21-c8bea25e5abb · 205 findings in all · rules version 5

Findings

Status: All · Open · Settled · Resolved

Severity: All · Medium · Low

State: All · Not yet reviewed · Under investigation · Explained · Confirmed error · Not a real disagreement

Showing findings that compare MutationAssessor. All models

9 findings match, most severe first.

FATHMM-MKL and MutationAssessor: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above FATHMM-MKL in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-MKL above MutationAssessor.

2 papers rank FATHMM-MKL above MutationAssessor in 19 of their 25 comparisons and MutationAssessor higher in 6. Livesey and Marsh 2025 ranks MutationAssessor above FATHMM-MKL in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-MKL or MutationAssessor higher2 other papers: FATHMM-MKL higher in 19, MutationAssessor higher in 6, 0 ties. Livesey and Marsh 2025 (disagrees): FATHMM-MKL higher in 0, MutationAssessor higher in 7, 0 ties.2 other papers19 FATHMM-MKL higher196 MutationAssessor higher6Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MutationAssessor higher7
FATHMM-MKL higherTieMutationAssessor higher
Compared
FATHMM-MKL, MutationAssessor

Not reviewed yet.

MetaSVM and MutationAssessor: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MetaSVM above MutationAssessor in 4 of 5 comparisons; 2 of 2 other studies rank MutationAssessor above MetaSVM.

3 papers rank MutationAssessor above MetaSVM in 22 of their 33 comparisons and MetaSVM higher in 10; 1 is a tie. Clinical likelihood ratios and balanced… ranks MetaSVM above MutationAssessor in 4 of its 6 comparisons and MutationAssessor higher in 1; 1 is a tie.

Comparisons in which each paper scores MutationAssessor or MetaSVM higher3 other papers: MutationAssessor higher in 22, MetaSVM higher in 10, 1 tie. Clinical likelihood ratios and balanced… (disagrees): MutationAssessor higher in 1, MetaSVM higher in 4, 1 tie.3 other papers22 MutationAssessor higher221 tie10 MetaSVM higher10Disagrees: Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes1 MutationAssessor higher11 tie14 MetaSVM higher4
MutationAssessor higherTieMetaSVM higher
Compared
MetaSVM, MutationAssessor

Not reviewed yet.

MPC and MutationAssessor: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above MPC in 6 of 7 comparisons; 2 of 2 other studies rank MPC above MutationAssessor.

2 papers rank MPC above MutationAssessor in all 16 of their comparisons. Livesey and Marsh 2025 ranks MutationAssessor above MPC in 6 of its 7 comparisons and MPC higher in 1.

Comparisons in which each paper scores MPC or MutationAssessor higher2 other papers: MPC higher in 16, MutationAssessor higher in 0, 0 ties. Livesey and Marsh 2025 (disagrees): MPC higher in 1, MutationAssessor higher in 6, 0 ties.2 other papers16 MPC higher16Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 MPC higher16 MutationAssessor higher6
MPC higherTieMutationAssessor higher
Compared
MPC, MutationAssessor

Not reviewed yet.

MutationAssessor and PROVEAN: An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks PROVEAN above MutationAssessor in 6 of 6 comparisons; 2 of 2 other studies rank MutationAssessor above PROVEAN.

5 papers rank MutationAssessor above PROVEAN in 21 of their 38 comparisons and PROVEAN higher in 15; 2 are ties. Of those papers, ProteinGym ranks PROVEAN higher on its own (1 of 1). An openly licensed benchmark… ranks PROVEAN above MutationAssessor in all 6 of its comparisons.

Comparisons in which each paper scores MutationAssessor or PROVEAN higher5 other papers: MutationAssessor higher in 21, PROVEAN higher in 15, 2 ties. An openly licensed benchmark… (disagrees): MutationAssessor higher in 0, PROVEAN higher in 6, 0 ties.5 other papers21 MutationAssessor higher212 tie15 PROVEAN higher15Disagrees: An openly licensed benchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers6 PROVEAN higher6
MutationAssessor higherTiePROVEAN higher
Compared
MutationAssessor, PROVEAN

Not reviewed yet.

FATHMM-XF and MutationAssessor: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above FATHMM-XF in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above MutationAssessor.

2 papers rank FATHMM-XF above MutationAssessor in 21 of their 26 comparisons and MutationAssessor higher in 5. Livesey and Marsh 2025 ranks MutationAssessor above FATHMM-XF in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or MutationAssessor higher2 other papers: FATHMM-XF higher in 21, MutationAssessor higher in 5, 0 ties. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, MutationAssessor higher in 7, 0 ties.2 other papers21 FATHMM-XF higher215 MutationAssessor higher5Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MutationAssessor higher7
FATHMM-XF higherTieMutationAssessor higher
Compared
FATHMM-XF, MutationAssessor

Not reviewed yet.

CADD and MutationAssessor: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above CADD in 6 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks CADD above MutationAssessor in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

4 papers rank CADD above MutationAssessor in 15 of their 28 comparisons and MutationAssessor higher in 13. Of those papers, Livesey and Marsh 2025 ranks MutationAssessor higher on its own (6 of 7).

Comparisons in which each paper scores CADD or MutationAssessor higher4 other papers: CADD higher in 15, MutationAssessor higher in 13, 0 ties.4 other papers15 CADD higher1513 MutationAssessor higher13
CADD higherTieMutationAssessor higher
Compared
CADD, MutationAssessor

Not reviewed yet.

MutationAssessor and MutFormer: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above MutationAssessor in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks MutationAssessor above MutFormer in 5 of 5 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank MutationAssessor above MutFormer in 5 of their 13 comparisons and MutFormer higher in 7; 1 is a tie. Of those papers, Livesey and Marsh 2025 ranks MutFormer higher on its own (7 of 7).

Comparisons in which each paper scores MutationAssessor or MutFormer higher2 other papers: MutationAssessor higher in 5, MutFormer higher in 7, 1 tie.2 other papers5 MutationAssessor higher51 tie7 MutFormer higher7
MutationAssessor higherTieMutFormer higher
Compared
MutationAssessor, MutFormer

Not reviewed yet.

MutationAssessor and PrimateAI: Comprehensive assessment of computational algorithms in predicting cancer driver mutations and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks PrimateAI above MutationAssessor in 19 of 20 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above PrimateAI in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

5 papers rank MutationAssessor above PrimateAI in 15 of their 40 comparisons and PrimateAI higher in 24; 1 is a tie. Of those papers, Comprehensive assessment… ranks PrimateAI higher on its own (19 of 20).

Comparisons in which each paper scores MutationAssessor or PrimateAI higher5 other papers: MutationAssessor higher in 15, PrimateAI higher in 24, 1 tie.5 other papers15 MutationAssessor higher151 tie24 PrimateAI higher24
MutationAssessor higherTiePrimateAI higher
Compared
MutationAssessor, PrimateAI

Not reviewed yet.

MutationAssessor and MVP: Comprehensive assessment of computational algorithms in predicting cancer driver mutations and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks MVP above MutationAssessor in 10 of 10 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above MVP in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

4 papers rank MutationAssessor above MVP in 11 of their 28 comparisons and MVP higher in 17. Of those papers, Comprehensive assessment…, An openly licensed benchmark… rank MVP higher on their own (10 of 10, 4 of 6).

Comparisons in which each paper scores MutationAssessor or MVP higher4 other papers: MutationAssessor higher in 11, MVP higher in 17, 0 ties.4 other papers11 MutationAssessor higher1117 MVP higher17
MutationAssessor higherTieMVP higher
Compared
MutationAssessor, MVP

Not reviewed yet.