rewirebio.iobenchmarks
Evidence and investigation

Contested results

Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.

These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.

A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.

How the check works, and how each follow-up is recorded: discrepancies between studies.

In this release

  • 121open
  • 5settled
  • 7resolved
  • 1medium severity
  • 132low severity

Release 2026-11-02-a7f03c9e55a5 · 133 findings in all · rules version 4

Findings

Status: All · Open · Settled · Resolved

Severity: All · Medium · Low

State: All · Not yet reviewed · Under investigation · Explained · Confirmed error · Not a real disagreement

121 findings match, most severe first.

ESM-2 and ESM3: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and pfmbench primary benchmark evidence disagree

  • Medium severity
  • Under investigation
  • two-study-conflict
  • Same task, different protocols

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM3 above ESM-2 in 18 of 18 comparisons; pfmbench primary benchmark evidence ranks ESM-2 above ESM3 in 9 of 12 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank ESM-2 above ESM3 in 9 of their 31 comparisons and ESM3 higher in 21; 1 is a tie. Of those papers, ProteinGym ranks ESM3 higher on its own (18 of 19).

Comparisons in which each paper scores ESM-2 or ESM3 higher2 other papers: ESM-2 higher in 9, ESM3 higher in 21, 1 tie.2 other papers9 ESM-2 higher91 tie21 ESM3 higher21
ESM-2 higherTieESM3 higher
Compared
ESM-2, ESM3

2 follow-up actions, latest 24 October 2026. Next check 6 November 2026

LRT and MetaLR: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks LRT above MetaLR in 15 of 17 comparisons; 2 of 2 other studies rank MetaLR above LRT.

2 papers rank MetaLR above LRT in 12 of their 13 comparisons and LRT higher in 1. Comprehensive assessment… ranks LRT above MetaLR in 15 of its 20 comparisons and MetaLR higher in 2; 3 are ties.

Comparisons in which each paper scores MetaLR or LRT higher2 other papers: MetaLR higher in 12, LRT higher in 1, 0 ties. Comprehensive assessment… (disagrees): MetaLR higher in 2, LRT higher in 15, 3 ties.2 other papers12 MetaLR higher121 LRT higherDisagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations2 MetaLR higher23 tie315 LRT higher15
MetaLR higherTieLRT higher
Compared
LRT, MetaLR

Not reviewed yet.

CADD and FATHMM-XF: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks FATHMM-XF above CADD in 10 of 10 comparisons; 2 of 2 other studies rank CADD above FATHMM-XF.

2 papers rank CADD above FATHMM-XF in all 12 comparisons that are not ties (1 is a tie). Comprehensive assessment… ranks FATHMM-XF above CADD in all 10 of its comparisons.

Comparisons in which each paper scores CADD or FATHMM-XF higher2 other papers: CADD higher in 12, FATHMM-XF higher in 0, 1 tie. Comprehensive assessment… (disagrees): CADD higher in 0, FATHMM-XF higher in 10, 0 ties.2 other papers12 CADD higher121 tieDisagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations10 FATHMM-XF higher10
CADD higherTieFATHMM-XF higher
Compared
CADD, FATHMM-XF

Not reviewed yet.

MVP and ESM-1b: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks MVP above ESM-1b in 2 of 2 comparisons; 2 of 2 other studies rank ESM-1b above MVP.

2 papers rank ESM-1b above MVP in all 13 of their comparisons. Brandes et al. 2023 ranks MVP above ESM-1b in all 2 comparisons that are not ties (2 are ties).

Comparisons in which each paper scores ESM-1b or MVP higher2 other papers: ESM-1b higher in 13, MVP higher in 0, 0 ties. Brandes et al. 2023 (disagrees): ESM-1b higher in 0, MVP higher in 2, 2 ties.2 other papers13 ESM-1b higher13Disagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods2 tie22 MVP higher2
ESM-1b higherTieMVP higher
Compared
MVP, ESM-1b

Not reviewed yet.

RNA-FM and RNAErnie: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks RNA-FM above RNAErnie in 28 of 36 comparisons; 2 of 2 other studies rank RNAErnie above RNA-FM.

2 papers rank RNAErnie above RNA-FM in all 6 comparisons that are not ties (13 are ties). Zero-shot benchmarking of RNA language… ranks RNA-FM above RNAErnie in 28 of its 36 comparisons and RNAErnie higher in 8.

Comparisons in which each paper scores RNAErnie or RNA-FM higher2 other papers: RNAErnie higher in 6, RNA-FM higher in 0, 13 ties. Zero-shot benchmarking of RNA language… (disagrees): RNAErnie higher in 8, RNA-FM higher in 28, 0 ties.2 other papers6 RNAErnie higher613 tie13Disagrees: Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning8 RNAErnie higher828 RNA-FM higher28
RNAErnie higherTieRNA-FM higher
Compared
RNA-FM, RNAErnie

Not reviewed yet.

DANN and GERP: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks GERP above DANN in 5 of 6 comparisons; 2 of 2 other studies rank DANN above GERP.

2 papers rank DANN above GERP in all 12 comparisons that are not ties (1 is a tie). Clinical likelihood ratios and balanced… ranks GERP above DANN in 5 of its 6 comparisons and DANN higher in 1.

Comparisons in which each paper scores DANN or GERP higher2 other papers: DANN higher in 12, GERP higher in 0, 1 tie. Clinical likelihood ratios and balanced… (disagrees): DANN higher in 1, GERP higher in 5, 0 ties.2 other papers12 DANN higher121 tieDisagrees: Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes1 DANN higher15 GERP higher5
DANN higherTieGERP higher
Compared
DANN, GERP

Not reviewed yet.

ClinPred and REVEL: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks REVEL above ClinPred in 6 of 7 comparisons; 2 of 2 other studies rank ClinPred above REVEL.

3 papers rank ClinPred above REVEL in 10 of their 12 comparisons and REVEL higher in 1; 1 is a tie. Livesey and Marsh 2025 ranks REVEL above ClinPred in 6 of its 7 comparisons and ClinPred higher in 1.

Comparisons in which each paper scores ClinPred or REVEL higher3 other papers: ClinPred higher in 10, REVEL higher in 1, 1 tie. Livesey and Marsh 2025 (disagrees): ClinPred higher in 1, REVEL higher in 6, 0 ties.3 other papers10 ClinPred higher101 tie1 REVEL higherDisagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 ClinPred higher16 REVEL higher6
ClinPred higherTieREVEL higher
Compared
ClinPred, REVEL

Not reviewed yet.

BayesDel and ESM-1b: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above BayesDel in 6 of 7 comparisons; 2 of 2 other studies rank BayesDel above ESM-1b.

2 papers rank BayesDel above ESM-1b in all 10 of their comparisons. Livesey and Marsh 2025 ranks ESM-1b above BayesDel in 6 of its 7 comparisons and BayesDel higher in 1.

Comparisons in which each paper scores BayesDel or ESM-1b higher2 other papers: BayesDel higher in 10, ESM-1b higher in 0, 0 ties. Livesey and Marsh 2025 (disagrees): BayesDel higher in 1, ESM-1b higher in 6, 0 ties.2 other papers10 BayesDel higher10Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 BayesDel higher16 ESM-1b higher6
BayesDel higherTieESM-1b higher
Compared
BayesDel, ESM-1b

Not reviewed yet.

MVP and REVEL: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MVP above REVEL in 4 of 5 comparisons; 3 of 3 other studies rank REVEL above MVP.

3 papers rank REVEL above MVP in all 19 comparisons that are not ties (4 are ties). Clinical likelihood ratios and balanced… ranks MVP above REVEL in 4 of its 5 comparisons and REVEL higher in 1.

Comparisons in which each paper scores REVEL or MVP higher3 other papers: REVEL higher in 19, MVP higher in 0, 4 ties. Clinical likelihood ratios and balanced… (disagrees): REVEL higher in 1, MVP higher in 4, 0 ties.3 other papers19 REVEL higher194 tie4Disagrees: Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes1 REVEL higher14 MVP higher4
REVEL higherTieMVP higher
Compared
MVP, REVEL

Not reviewed yet.

ESM-1b and EVE: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Under investigation
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks ESM-1b above EVE in 6 of 6 comparisons; 2 of 2 other studies rank EVE above ESM-1b.

2 papers rank EVE above ESM-1b in 22 of their 26 comparisons and ESM-1b higher in 1; 3 are ties. Brandes et al. 2023 ranks ESM-1b above EVE in all 6 comparisons that are not ties (2 are ties).

Comparisons in which each paper scores EVE or ESM-1b higher2 other papers: EVE higher in 22, ESM-1b higher in 1, 3 ties. Brandes et al. 2023 (disagrees): EVE higher in 0, ESM-1b higher in 6, 2 ties.2 other papers22 EVE higher223 tie31 ESM-1b higherDisagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods2 tie26 ESM-1b higher6
EVE higherTieESM-1b higher
Compared
ESM-1b, EVE

1 follow-up action, latest 23 October 2026. Next check 6 November 2026

FATHMM-MKL and MutationAssessor: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above FATHMM-MKL in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-MKL above MutationAssessor.

2 papers rank FATHMM-MKL above MutationAssessor in 19 of their 25 comparisons and MutationAssessor higher in 6. Livesey and Marsh 2025 ranks MutationAssessor above FATHMM-MKL in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-MKL or MutationAssessor higher2 other papers: FATHMM-MKL higher in 19, MutationAssessor higher in 6, 0 ties. Livesey and Marsh 2025 (disagrees): FATHMM-MKL higher in 0, MutationAssessor higher in 7, 0 ties.2 other papers19 FATHMM-MKL higher196 MutationAssessor higher6Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MutationAssessor higher7
FATHMM-MKL higherTieMutationAssessor higher
Compared
FATHMM-MKL, MutationAssessor

Not reviewed yet.

GenoCanyon and SIFT: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks GenoCanyon above SIFT in 6 of 8 comparisons; 3 of 3 other studies rank SIFT above GenoCanyon.

3 papers rank SIFT above GenoCanyon in 14 of their 16 comparisons and GenoCanyon higher in 1; 1 is a tie. Comprehensive assessment… ranks GenoCanyon above SIFT in 6 of its 10 comparisons and SIFT higher in 2; 2 are ties.

Comparisons in which each paper scores SIFT or GenoCanyon higher3 other papers: SIFT higher in 14, GenoCanyon higher in 1, 1 tie. Comprehensive assessment… (disagrees): SIFT higher in 2, GenoCanyon higher in 6, 2 ties.3 other papers14 SIFT higher141 tie1 GenoCanyon higherDisagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations2 SIFT higher22 tie26 GenoCanyon higher6
SIFT higherTieGenoCanyon higher
Compared
GenoCanyon, SIFT

Not reviewed yet.

MetaLR and ESM-1b: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks MetaLR above ESM-1b in 2 of 2 comparisons; 2 of 2 other studies rank ESM-1b above MetaLR.

2 papers rank ESM-1b above MetaLR in all 13 of their comparisons. Brandes et al. 2023 ranks MetaLR above ESM-1b in all 2 comparisons that are not ties (2 are ties).

Comparisons in which each paper scores ESM-1b or MetaLR higher2 other papers: ESM-1b higher in 13, MetaLR higher in 0, 0 ties. Brandes et al. 2023 (disagrees): ESM-1b higher in 0, MetaLR higher in 2, 2 ties.2 other papers13 ESM-1b higher13Disagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods2 tie22 MetaLR higher2
ESM-1b higherTieMetaLR higher
Compared
MetaLR, ESM-1b

Not reviewed yet.

MetaSVM and ESM-1b: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks MetaSVM above ESM-1b in 2 of 2 comparisons; 2 of 2 other studies rank ESM-1b above MetaSVM.

2 papers rank ESM-1b above MetaSVM in all 13 of their comparisons. Brandes et al. 2023 ranks MetaSVM above ESM-1b in all 2 comparisons that are not ties (2 are ties).

Comparisons in which each paper scores ESM-1b or MetaSVM higher2 other papers: ESM-1b higher in 13, MetaSVM higher in 0, 0 ties. Brandes et al. 2023 (disagrees): ESM-1b higher in 0, MetaSVM higher in 2, 2 ties.2 other papers13 ESM-1b higher13Disagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods2 tie22 MetaSVM higher2
ESM-1b higherTieMetaSVM higher
Compared
MetaSVM, ESM-1b

Not reviewed yet.

FATHMM-XF and M-CAP: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks M-CAP above FATHMM-XF in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above M-CAP.

2 papers rank FATHMM-XF above M-CAP in 22 of their 26 comparisons and M-CAP higher in 3; 1 is a tie. Livesey and Marsh 2025 ranks M-CAP above FATHMM-XF in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or M-CAP higher2 other papers: FATHMM-XF higher in 22, M-CAP higher in 3, 1 tie. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, M-CAP higher in 7, 0 ties.2 other papers22 FATHMM-XF higher221 tie3 M-CAP higher3Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 M-CAP higher7
FATHMM-XF higherTieM-CAP higher
Compared
FATHMM-XF, M-CAP

Not reviewed yet.

FATHMM-MKL and MetaLR: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above FATHMM-MKL in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-MKL above MetaLR.

2 papers rank FATHMM-MKL above MetaLR in 18 of their 25 comparisons and MetaLR higher in 5; 2 are ties. Livesey and Marsh 2025 ranks MetaLR above FATHMM-MKL in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-MKL or MetaLR higher2 other papers: FATHMM-MKL higher in 18, MetaLR higher in 5, 2 ties. Livesey and Marsh 2025 (disagrees): FATHMM-MKL higher in 0, MetaLR higher in 7, 0 ties.2 other papers18 FATHMM-MKL higher182 tie5 MetaLR higher5Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MetaLR higher7
FATHMM-MKL higherTieMetaLR higher
Compared
FATHMM-MKL, MetaLR

Not reviewed yet.

FATHMM-XF and MetaLR: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above FATHMM-XF in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above MetaLR.

2 papers rank FATHMM-XF above MetaLR in 22 of their 26 comparisons and MetaLR higher in 3; 1 is a tie. Livesey and Marsh 2025 ranks MetaLR above FATHMM-XF in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or MetaLR higher2 other papers: FATHMM-XF higher in 22, MetaLR higher in 3, 1 tie. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, MetaLR higher in 7, 0 ties.2 other papers22 FATHMM-XF higher221 tie3 MetaLR higher3Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MetaLR higher7
FATHMM-XF higherTieMetaLR higher
Compared
FATHMM-XF, MetaLR

Not reviewed yet.

FATHMM-XF and MetaSVM: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaSVM above FATHMM-XF in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above MetaSVM.

2 papers rank FATHMM-XF above MetaSVM in 22 of their 26 comparisons and MetaSVM higher in 4. Livesey and Marsh 2025 ranks MetaSVM above FATHMM-XF in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or MetaSVM higher2 other papers: FATHMM-XF higher in 22, MetaSVM higher in 4, 0 ties. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, MetaSVM higher in 7, 0 ties.2 other papers22 FATHMM-XF higher224 MetaSVM higher4Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MetaSVM higher7
FATHMM-XF higherTieMetaSVM higher
Compared
FATHMM-XF, MetaSVM

Not reviewed yet.

REVEL and BayesDel: An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks BayesDel above REVEL in 6 of 6 comparisons; 2 of 2 other studies rank REVEL above BayesDel.

3 papers rank REVEL above BayesDel in 11 of their 14 comparisons and BayesDel higher in 2; 1 is a tie. Of those papers, ProteinGym ranks BayesDel higher on its own (1 of 1). An openly licensed benchmark… ranks BayesDel above REVEL in all 6 of its comparisons.

Comparisons in which each paper scores REVEL or BayesDel higher3 other papers: REVEL higher in 11, BayesDel higher in 2, 1 tie. An openly licensed benchmark… (disagrees): REVEL higher in 0, BayesDel higher in 6, 0 ties.3 other papers11 REVEL higher111 tie2 BayesDel higher2Disagrees: An openly licensed benchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers6 BayesDel higher6
REVEL higherTieBayesDel higher
Compared
REVEL, BayesDel

Not reviewed yet.

LRT and MetaSVM: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks LRT above MetaSVM in 16 of 20 comparisons; 2 of 2 other studies rank MetaSVM above LRT.

2 papers rank MetaSVM above LRT in 12 of their 13 comparisons and LRT higher in 1. Comprehensive assessment… ranks LRT above MetaSVM in 16 of its 20 comparisons and MetaSVM higher in 4.

Comparisons in which each paper scores MetaSVM or LRT higher2 other papers: MetaSVM higher in 12, LRT higher in 1, 0 ties. Comprehensive assessment… (disagrees): MetaSVM higher in 4, LRT higher in 16, 0 ties.2 other papers12 MetaSVM higher121 LRT higherDisagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations4 MetaSVM higher416 LRT higher16
MetaSVM higherTieLRT higher
Compared
LRT, MetaSVM

Not reviewed yet.

Nucleotide Transformer v2 and Nucleotide Transformer: GENEB leaderboard (leaderboard.json, 13 task-type aggregates) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

GENEB leaderboard (leaderboard.json, 13 task-type aggregates) ranks Nucleotide Transformer above Nucleotide Transformer v2 in 37 of 44 comparisons; 2 of 2 other studies rank Nucleotide Transformer v2 above Nucleotide Transformer.

2 papers rank Nucleotide Transformer v2 above Nucleotide Transformer in 33 of their 46 comparisons and Nucleotide Transformer higher in 7; 6 are ties. GENEB leaderboard ranks Nucleotide Transformer above Nucleotide Transformer v2 in 37 of its 45 comparisons and Nucleotide Transformer v2 higher in 7; 1 is a tie.

Comparisons in which each paper scores Nucleotide Transformer v2 or Nucleotide Transformer higher2 other papers: Nucleotide Transformer v2 higher in 33, Nucleotide Transformer higher in 7, 6 ties. GENEB leaderboard (disagrees): Nucleotide Transformer v2 higher in 7, Nucleotide Transformer higher in 37, 1 tie.2 other papers33 Nucleotide Transformer v2 higher336 tie67 Nucleotide Transformer higher7Disagrees: GENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)7 Nucleotide Transformer v2 higher71 tie37 Nucleotide Transformer higher37
Nucleotide Transformer v2 higherTieNucleotide Transformer higher
Compared
Nucleotide Transformer v2, Nucleotide Transformer

Not reviewed yet.

GPN-MSA and phyloP: TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) ranks phyloP above GPN-MSA in 2 of 2 comparisons; 2 of 2 other studies rank GPN-MSA above phyloP.

3 papers rank GPN-MSA above phyloP in 59 of their 67 comparisons and phyloP higher in 2; 6 are ties. TraitGym official metrics ranks phyloP above GPN-MSA in all 2 of its comparisons.

Comparisons in which each paper scores GPN-MSA or phyloP higher3 other papers: GPN-MSA higher in 59, phyloP higher in 2, 6 ties. TraitGym official metrics (disagrees): GPN-MSA higher in 0, phyloP higher in 2, 0 ties.3 other papers59 GPN-MSA higher596 tie2 phyloP higherDisagrees: TraitGym official metricsTraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)2 phyloP higher2
GPN-MSA higherTiephyloP higher
Compared
GPN-MSA, phyloP

Not reviewed yet.

MutPred and ESM-1b: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above MutPred in 6 of 7 comparisons; 2 of 2 other studies rank MutPred above ESM-1b.

3 papers rank MutPred above ESM-1b in 8 of their 11 comparisons and ESM-1b higher in 1; 2 are ties. Of those papers, ProteinGym ranks ESM-1b higher on its own (1 of 1). Livesey and Marsh 2025 ranks ESM-1b above MutPred in 6 of its 7 comparisons and MutPred higher in 1.

Comparisons in which each paper scores MutPred or ESM-1b higher3 other papers: MutPred higher in 8, ESM-1b higher in 1, 2 ties. Livesey and Marsh 2025 (disagrees): MutPred higher in 1, ESM-1b higher in 6, 0 ties.3 other papers8 MutPred higher82 tie21 ESM-1b higher1Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 MutPred higher16 ESM-1b higher6
MutPred higherTieESM-1b higher
Compared
MutPred, ESM-1b

Not reviewed yet.

MPC and MutationAssessor: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above MPC in 6 of 7 comparisons; 2 of 2 other studies rank MPC above MutationAssessor.

2 papers rank MPC above MutationAssessor in all 16 of their comparisons. Livesey and Marsh 2025 ranks MutationAssessor above MPC in 6 of its 7 comparisons and MPC higher in 1.

Comparisons in which each paper scores MPC or MutationAssessor higher2 other papers: MPC higher in 16, MutationAssessor higher in 0, 0 ties. Livesey and Marsh 2025 (disagrees): MPC higher in 1, MutationAssessor higher in 6, 0 ties.2 other papers16 MPC higher16Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 MPC higher16 MutationAssessor higher6
MPC higherTieMutationAssessor higher
Compared
MPC, MutationAssessor

Not reviewed yet.

MVP and PolyPhen-2: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks PolyPhen-2 above MVP in 6 of 6 comparisons; 3 of 3 other studies rank MVP above PolyPhen-2.

3 papers rank MVP above PolyPhen-2 in 18 of their 21 comparisons and PolyPhen-2 higher in 2; 1 is a tie. Livesey and Marsh 2025 ranks PolyPhen-2 above MVP in all 6 of its comparisons.

Comparisons in which each paper scores MVP or PolyPhen-2 higher3 other papers: MVP higher in 18, PolyPhen-2 higher in 2, 1 tie. Livesey and Marsh 2025 (disagrees): MVP higher in 0, PolyPhen-2 higher in 6, 0 ties.3 other papers18 MVP higher181 tie2 PolyPhen-2 higher2Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)6 PolyPhen-2 higher6
MVP higherTiePolyPhen-2 higher
Compared
MVP, PolyPhen-2

Not reviewed yet.

GenoCanyon and phyloP: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks GenoCanyon above phyloP in 3 of 3 comparisons; 2 of 2 other studies rank phyloP above GenoCanyon.

2 papers rank phyloP above GenoCanyon in 11 of their 13 comparisons and GenoCanyon higher in 1; 1 is a tie. Brandes et al. 2023 ranks GenoCanyon above phyloP in all 3 comparisons that are not ties (1 is a tie).

Comparisons in which each paper scores phyloP or GenoCanyon higher2 other papers: phyloP higher in 11, GenoCanyon higher in 1, 1 tie. Brandes et al. 2023 (disagrees): phyloP higher in 0, GenoCanyon higher in 3, 1 tie.2 other papers11 phyloP higher111 tie1 GenoCanyon higherDisagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods1 tie13 GenoCanyon higher3
phyloP higherTieGenoCanyon higher
Compared
GenoCanyon, phyloP

Not reviewed yet.

VEST4 and ESM-1b: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above VEST4 in 6 of 7 comparisons; 2 of 2 other studies rank VEST4 above ESM-1b.

2 papers rank VEST4 above ESM-1b in 7 of their 10 comparisons and ESM-1b higher in 1; 2 are ties. Livesey and Marsh 2025 ranks ESM-1b above VEST4 in 6 of its 7 comparisons and VEST4 higher in 1.

Comparisons in which each paper scores VEST4 or ESM-1b higher2 other papers: VEST4 higher in 7, ESM-1b higher in 1, 2 ties. Livesey and Marsh 2025 (disagrees): VEST4 higher in 1, ESM-1b higher in 6, 0 ties.2 other papers7 VEST4 higher72 tie21 ESM-1b higher1Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 VEST4 higher16 ESM-1b higher6
VEST4 higherTieESM-1b higher
Compared
VEST4, ESM-1b

Not reviewed yet.

FATHMM-XF and MutationAssessor: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above FATHMM-XF in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above MutationAssessor.

2 papers rank FATHMM-XF above MutationAssessor in 21 of their 26 comparisons and MutationAssessor higher in 5. Livesey and Marsh 2025 ranks MutationAssessor above FATHMM-XF in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or MutationAssessor higher2 other papers: FATHMM-XF higher in 21, MutationAssessor higher in 5, 0 ties. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, MutationAssessor higher in 7, 0 ties.2 other papers21 FATHMM-XF higher215 MutationAssessor higher5Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MutationAssessor higher7
FATHMM-XF higherTieMutationAssessor higher
Compared
FATHMM-XF, MutationAssessor

Not reviewed yet.

GenoCanyon and MetaSVM: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks GenoCanyon above MetaSVM in 6 of 8 comparisons; 2 of 2 other studies rank MetaSVM above GenoCanyon.

2 papers rank MetaSVM above GenoCanyon in 12 of their 13 comparisons and GenoCanyon higher in 1. Comprehensive assessment… ranks GenoCanyon above MetaSVM in 6 of its 10 comparisons and MetaSVM higher in 2; 2 are ties.

Comparisons in which each paper scores MetaSVM or GenoCanyon higher2 other papers: MetaSVM higher in 12, GenoCanyon higher in 1, 0 ties. Comprehensive assessment… (disagrees): MetaSVM higher in 2, GenoCanyon higher in 6, 2 ties.2 other papers12 MetaSVM higher121 GenoCanyon higherDisagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations2 MetaSVM higher22 tie26 GenoCanyon higher6
MetaSVM higherTieGenoCanyon higher
Compared
GenoCanyon, MetaSVM

Not reviewed yet.

SiPhy and phyloP: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks SiPhy above phyloP in 3 of 4 comparisons; 2 of 2 other studies rank phyloP above SiPhy.

2 papers rank phyloP above SiPhy in all 13 of their comparisons. Brandes et al. 2023 ranks SiPhy above phyloP in 3 of its 4 comparisons and phyloP higher in 1.

Comparisons in which each paper scores phyloP or SiPhy higher2 other papers: phyloP higher in 13, SiPhy higher in 0, 0 ties. Brandes et al. 2023 (disagrees): phyloP higher in 1, SiPhy higher in 3, 0 ties.2 other papers13 phyloP higher13Disagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods1 phyloP higher13 SiPhy higher3
phyloP higherTieSiPhy higher
Compared
SiPhy, phyloP

Not reviewed yet.

Geneformer and scGPT: Kedzierska et al. 2025, Additional file 1 (Supplementary Tables S1-S7, xlsx) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Kedzierska et al. 2025, Additional file 1 (Supplementary Tables S1-S7, xlsx) ranks scGPT above Geneformer in 83 of 89 comparisons; 2 of 2 other studies rank Geneformer above scGPT.

3 papers rank Geneformer above scGPT in 5 of their 7 comparisons and scGPT higher in 2. Of those papers, Parameter-Efficient Fine-Tuning… ranks scGPT higher on its own (1 of 1). Kedzierska et al. 2025 ranks scGPT above Geneformer in 83 of its 97 comparisons and Geneformer higher in 6; 8 are ties.

Comparisons in which each paper scores Geneformer or scGPT higher3 other papers: Geneformer higher in 5, scGPT higher in 2, 0 ties. Kedzierska et al. 2025 (disagrees): Geneformer higher in 6, scGPT higher in 83, 8 ties.3 other papers5 Geneformer higher52 scGPT higher2Disagrees: Kedzierska et al. 2025Kedzierska et al. 2025, Additional file 1 (Supplementary Tables S1-S7, xlsx)6 Geneformer higher8 tie83 scGPT higher83
Geneformer higherTiescGPT higher
Compared
Geneformer, scGPT

Not reviewed yet.

FATHMM-XF and SIFT: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks SIFT above FATHMM-XF in 6 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above SIFT.

2 papers rank FATHMM-XF above SIFT in 25 of their 26 comparisons and SIFT higher in 1. Livesey and Marsh 2025 ranks SIFT above FATHMM-XF in 6 of its 7 comparisons and FATHMM-XF higher in 1.

Comparisons in which each paper scores FATHMM-XF or SIFT higher2 other papers: FATHMM-XF higher in 25, SIFT higher in 1, 0 ties. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 1, SIFT higher in 6, 0 ties.2 other papers25 FATHMM-XF higher251 SIFT higherDisagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 FATHMM-XF higher16 SIFT higher6
FATHMM-XF higherTieSIFT higher
Compared
FATHMM-XF, SIFT

Not reviewed yet.

FATHMM-XF and PolyPhen-2: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks PolyPhen-2 above FATHMM-XF in 6 of 6 comparisons; 2 of 2 other studies rank FATHMM-XF above PolyPhen-2.

2 papers rank FATHMM-XF above PolyPhen-2 in 20 of their 26 comparisons and PolyPhen-2 higher in 1; 5 are ties. Livesey and Marsh 2025 ranks PolyPhen-2 above FATHMM-XF in all 6 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or PolyPhen-2 higher2 other papers: FATHMM-XF higher in 20, PolyPhen-2 higher in 1, 5 ties. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, PolyPhen-2 higher in 6, 0 ties.2 other papers20 FATHMM-XF higher205 tie51 PolyPhen-2 higherDisagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)6 PolyPhen-2 higher6
FATHMM-XF higherTiePolyPhen-2 higher
Compared
FATHMM-XF, PolyPhen-2

Not reviewed yet.

SiPhy and GERP: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks SiPhy above GERP in 4 of 4 comparisons; 2 of 2 other studies rank GERP above SiPhy.

2 papers rank GERP above SiPhy in 11 of their 13 comparisons and SiPhy higher in 1; 1 is a tie. Brandes et al. 2023 ranks SiPhy above GERP in all 4 of its comparisons.

Comparisons in which each paper scores GERP or SiPhy higher2 other papers: GERP higher in 11, SiPhy higher in 1, 1 tie. Brandes et al. 2023 (disagrees): GERP higher in 0, SiPhy higher in 4, 0 ties.2 other papers11 GERP higher111 tie1 SiPhy higherDisagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods4 SiPhy higher4
GERP higherTieSiPhy higher
Compared
SiPhy, GERP

Not reviewed yet.

DEOGEN2 and MutFormer: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above DEOGEN2 in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks DEOGEN2 above MutFormer in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank DEOGEN2 above MutFormer in 6 of their 13 comparisons and MutFormer higher in 7. Of those papers, Livesey and Marsh 2025 ranks MutFormer higher on its own (7 of 7).

Comparisons in which each paper scores DEOGEN2 or MutFormer higher2 other papers: DEOGEN2 higher in 6, MutFormer higher in 7, 0 ties.2 other papers6 DEOGEN2 higher67 MutFormer higher7
DEOGEN2 higherTieMutFormer higher
Compared
DEOGEN2, MutFormer

Not reviewed yet.

CARP and VESPAl: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks VESPAl above CARP in 15 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above VESPAl in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank CARP above VESPAl in 10 of their 25 comparisons and VESPAl higher in 15. Of those papers, ProteinGym ranks VESPAl higher on its own (15 of 19).

Comparisons in which each paper scores CARP or VESPAl higher2 other papers: CARP higher in 10, VESPAl higher in 15, 0 ties.2 other papers10 CARP higher1015 VESPAl higher15
CARP higherTieVESPAl higher
Compared
CARP, VESPAl

Not reviewed yet.

SuSPect and MetaLR: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MetaLR above SuSPect in 4 of 5 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks SuSPect above MetaLR in 6 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank SuSPect above MetaLR in 7 of their 13 comparisons and MetaLR higher in 5; 1 is a tie. Of those papers, Clinical likelihood ratios and balanced… ranks MetaLR higher on its own (4 of 6).

Comparisons in which each paper scores SuSPect or MetaLR higher2 other papers: SuSPect higher in 7, MetaLR higher in 5, 1 tie.2 other papers7 SuSPect higher71 tie5 MetaLR higher5
SuSPect higherTieMetaLR higher
Compared
SuSPect, MetaLR

Not reviewed yet.

MetaRNN and VARITY: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks VARITY above MetaRNN in 6 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks MetaRNN above VARITY in 3 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

3 papers rank MetaRNN above VARITY in 5 of their 14 comparisons and VARITY higher in 7; 2 are ties. Of those papers, Livesey and Marsh 2025 ranks VARITY higher on its own (6 of 7).

Comparisons in which each paper scores MetaRNN or VARITY higher3 other papers: MetaRNN higher in 5, VARITY higher in 7, 2 ties.3 other papers5 MetaRNN higher52 tie27 VARITY higher7
MetaRNN higherTieVARITY higher
Compared
MetaRNN, VARITY

Not reviewed yet.

MPC and MutFormer: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above MPC in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks MPC above MutFormer in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank MPC above MutFormer in 6 of their 13 comparisons and MutFormer higher in 7. Of those papers, Livesey and Marsh 2025 ranks MutFormer higher on its own (7 of 7).

Comparisons in which each paper scores MPC or MutFormer higher2 other papers: MPC higher in 6, MutFormer higher in 7, 0 ties.2 other papers6 MPC higher67 MutFormer higher7
MPC higherTieMutFormer higher
Compared
MPC, MutFormer

Not reviewed yet.

ESM-1b and Tranception: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above ESM-1b in 15 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above Tranception in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank ESM-1b above Tranception in 11 of their 26 comparisons and Tranception higher in 15. Of those papers, ProteinGym ranks Tranception higher on its own (15 of 19).

Comparisons in which each paper scores ESM-1b or Tranception higher2 other papers: ESM-1b higher in 11, Tranception higher in 15, 0 ties.2 other papers11 ESM-1b higher1115 Tranception higher15
ESM-1b higherTieTranception higher
Compared
ESM-1b, Tranception

Not reviewed yet.

SNAP2 and FATHMM-MKL: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks FATHMM-MKL above SNAP2 in 3 of 4 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks SNAP2 above FATHMM-MKL in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank SNAP2 above FATHMM-MKL in 8 of their 12 comparisons and FATHMM-MKL higher in 3; 1 is a tie. Of those papers, Clinical likelihood ratios and balanced… ranks FATHMM-MKL higher on its own (3 of 5).

Comparisons in which each paper scores SNAP2 or FATHMM-MKL higher2 other papers: SNAP2 higher in 8, FATHMM-MKL higher in 3, 1 tie.2 other papers8 SNAP2 higher81 tie3 FATHMM-MKL higher3
SNAP2 higherTieFATHMM-MKL higher
Compared
SNAP2, FATHMM-MKL

Not reviewed yet.

Nucleotide Transformer and RNAErnie: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks Nucleotide Transformer above RNAErnie in 7 of 7 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above Nucleotide Transformer in 4 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank Nucleotide Transformer above RNAErnie in 7 of their 19 comparisons and RNAErnie higher in 4; 8 are ties. Of those papers, RNAGym fitness leaderboard ranks RNAErnie higher on its own (4 of 4).

Comparisons in which each paper scores Nucleotide Transformer or RNAErnie higher2 other papers: Nucleotide Transformer higher in 7, RNAErnie higher in 4, 8 ties.2 other papers7 Nucleotide Transformer higher78 tie84 RNAErnie higher4
Nucleotide Transformer higherTieRNAErnie higher
Compared
Nucleotide Transformer, RNAErnie

Not reviewed yet.

DNABERT-2 and PhyloGPN: GUANinE v1.1 reveals complementarity of supervised and genomic language models and Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks DNABERT-2 above PhyloGPN in 6 of 6 comparisons; Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) ranks PhyloGPN above DNABERT-2 in 14 of 14 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank DNABERT-2 above PhyloGPN in 6 of their 20 comparisons and PhyloGPN higher in 14. Of those papers, Genomic heterogeneity inflates… ranks PhyloGPN higher on its own (14 of 14).

Comparisons in which each paper scores DNABERT-2 or PhyloGPN higher2 other papers: DNABERT-2 higher in 6, PhyloGPN higher in 14, 0 ties.2 other papers6 DNABERT-2 higher614 PhyloGPN higher14
DNABERT-2 higherTiePhyloGPN higher
Compared
DNABERT-2, PhyloGPN

Not reviewed yet.

MutFormer and SIFT: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above SIFT in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks SIFT above MutFormer in 3 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank MutFormer above SIFT in 8 of their 13 comparisons and SIFT higher in 3; 2 are ties. Of those papers, An openly licensed benchmark… ranks SIFT higher on its own (3 of 6).

Comparisons in which each paper scores MutFormer or SIFT higher2 other papers: MutFormer higher in 8, SIFT higher in 3, 2 ties.2 other papers8 MutFormer higher82 tie23 SIFT higher3
MutFormer higherTieSIFT higher
Compared
MutFormer, SIFT

Not reviewed yet.

DNABERT and RNA-MSM: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks DNABERT above RNA-MSM in 4 of 4 comparisons; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks RNA-MSM above DNABERT in 8 of 8 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank DNABERT above RNA-MSM in 4 of their 20 comparisons and RNA-MSM higher in 8; 8 are ties. Of those papers, Zero-shot benchmarking of RNA language… ranks RNA-MSM higher on its own (8 of 8).

Comparisons in which each paper scores DNABERT or RNA-MSM higher2 other papers: DNABERT higher in 4, RNA-MSM higher in 8, 8 ties.2 other papers4 DNABERT higher48 tie88 RNA-MSM higher8
DNABERT higherTieRNA-MSM higher
Compared
DNABERT, RNA-MSM

Not reviewed yet.

ChromBPNet and Caduceus: GUANinE v1.1 reveals complementarity of supervised and genomic language models and Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks Caduceus above ChromBPNet in 6 of 6 comparisons; Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants ranks ChromBPNet above Caduceus in 4 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank ChromBPNet above Caduceus in 4 of their 10 comparisons and Caduceus higher in 6. Of those papers, GUANinE v1.1 reveals complementarity… ranks Caduceus higher on its own (6 of 6).

Comparisons in which each paper scores ChromBPNet or Caduceus higher2 other papers: ChromBPNet higher in 4, Caduceus higher in 6, 0 ties.2 other papers4 ChromBPNet higher46 Caduceus higher6
ChromBPNet higherTieCaduceus higher
Compared
ChromBPNet, Caduceus

Not reviewed yet.

CARP and VESPA: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks VESPA above CARP in 18 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above VESPA in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank CARP above VESPA in 7 of their 25 comparisons and VESPA higher in 18. Of those papers, ProteinGym ranks VESPA higher on its own (18 of 19).

Comparisons in which each paper scores CARP or VESPA higher2 other papers: CARP higher in 7, VESPA higher in 18, 0 ties.2 other papers7 CARP higher718 VESPA higher18
CARP higherTieVESPA higher
Compared
CARP, VESPA

Not reviewed yet.

Condel and MetaSVM: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MetaSVM above Condel in 4 of 5 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks Condel above MetaSVM in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank Condel above MetaSVM in 8 of their 13 comparisons and MetaSVM higher in 4; 1 is a tie. Of those papers, Clinical likelihood ratios and balanced… ranks MetaSVM higher on its own (4 of 6).

Comparisons in which each paper scores Condel or MetaSVM higher2 other papers: Condel higher in 8, MetaSVM higher in 4, 1 tie.2 other papers8 Condel higher81 tie4 MetaSVM higher4
Condel higherTieMetaSVM higher
Compared
Condel, MetaSVM

Not reviewed yet.

ESM-2 and Tranception: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above ESM-2 in 16 of 18 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-2 above Tranception in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank ESM-2 above Tranception in 9 of their 26 comparisons and Tranception higher in 16; 1 is a tie. Of those papers, ProteinGym ranks Tranception higher on its own (16 of 19).

Comparisons in which each paper scores ESM-2 or Tranception higher2 other papers: ESM-2 higher in 9, Tranception higher in 16, 1 tie.2 other papers9 ESM-2 higher91 tie16 Tranception higher16
ESM-2 higherTieTranception higher
Compared
ESM-2, Tranception

Not reviewed yet.

ESM-1v and DeepSequence: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks DeepSequence above ESM-1v in 13 of 17 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1v above DeepSequence in 6 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank ESM-1v above DeepSequence in 10 of their 26 comparisons and DeepSequence higher in 14; 2 are ties. Of those papers, ProteinGym ranks DeepSequence higher on its own (13 of 19).

Comparisons in which each paper scores ESM-1v or DeepSequence higher2 other papers: ESM-1v higher in 10, DeepSequence higher in 14, 2 ties.2 other papers10 ESM-1v higher102 tie14 DeepSequence higher14
ESM-1v higherTieDeepSequence higher
Compared
ESM-1v, DeepSequence

Not reviewed yet.

Next page