rewirebio.iobenchmarks
Evidence and investigation

Contested results

Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.

These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.

A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.

How the check works, and how each follow-up is recorded: discrepancies between studies.

In this release

  • 192open
  • 6settled
  • 7resolved
  • 3medium severity
  • 202low severity

Release 2026-11-22-bbed7dce2323 · 205 findings in all · rules version 5

Findings

Status: All · Open · Settled · Resolved

Severity: All · Medium · Low

State: All · Not yet reviewed · Under investigation · Explained · Confirmed error · Not a real disagreement

192 findings match, most severe first.

GRNBoost2 and PPCOR (partial Pearson correlation): BEELINE: 10.1038/s41592-019-0690-6 source data published 2020 reverses the order

  • Medium severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

BEELINE: 10.1038/s41592-019-0690-6 source data published 2020 ranks PPCOR (partial Pearson correlation) above GRNBoost2 in 9 of 12 comparisons; 2 of 2 other studies rank GRNBoost2 above PPCOR (partial Pearson correlation).

2 papers rank GRNBoost2 above PPCOR (partial Pearson correlation) in 47 of their 59 comparisons and PPCOR (partial Pearson correlation) higher in 8; 4 are ties. BEELINE ranks PPCOR (partial Pearson correlation) above GRNBoost2 in 9 of its 12 comparisons and GRNBoost2 higher in 3.

Comparisons in which each paper scores GRNBoost2 or PPCOR (partial Pearson correlation) higher2 other papers: GRNBoost2 higher in 47, PPCOR (partial Pearson correlation) higher in 8, 4 ties. BEELINE (disagrees): GRNBoost2 higher in 3, PPCOR (partial Pearson correlation) higher in 9, 0 ties.2 other papers47 GRNBoost2 higher474 tie8 PPCOR (partial Pearson correlation) higher8Disagrees: BEELINEBEELINE: 10.1038/s41592-019-0690-6 source data published 20203 GRNBoost2 higher39 PPCOR (partial Pearson correlation) higher9
GRNBoost2 higherTiePPCOR (partial Pearson correlation) higher
Compared
GRNBoost2, PPCOR (partial Pearson correlation)

Not reviewed yet.

PROVEAN and SIFT: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Medium severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks SIFT above PROVEAN in 3 of 3 comparisons; 3 of 3 other studies rank PROVEAN above SIFT.

5 papers rank PROVEAN above SIFT in 32 of their 39 comparisons and SIFT higher in 4; 3 are ties. Brandes et al. 2023 ranks SIFT above PROVEAN in all 3 comparisons that are not ties (1 is a tie).

Comparisons in which each paper scores PROVEAN or SIFT higher5 other papers: PROVEAN higher in 32, SIFT higher in 4, 3 ties. Brandes et al. 2023 (disagrees): PROVEAN higher in 0, SIFT higher in 3, 1 tie.5 other papers32 PROVEAN higher323 tie4 SIFT higher4Disagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods1 tie13 SIFT higher3
PROVEAN higherTieSIFT higher
Compared
PROVEAN, SIFT

Not reviewed yet.

ESM-2 and ESM3: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and pfmbench primary benchmark evidence disagree

  • Medium severity
  • Under investigation
  • Two-study conflict
  • Same task, different protocols

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM3 above ESM-2 in 18 of 18 comparisons; pfmbench primary benchmark evidence ranks ESM-2 above ESM3 in 9 of 12 comparisons. No other study gives a verdict, so neither is the outlier.

3 papers rank ESM-2 above ESM3 in 20 of their 59 comparisons and ESM3 higher in 32; 7 are ties. Of those papers, ProteinGym ranks ESM3 higher on its own (18 of 19).

Comparisons in which each paper scores ESM-2 or ESM3 higher3 other papers: ESM-2 higher in 20, ESM3 higher in 32, 7 ties.3 other papers20 ESM-2 higher207 tie732 ESM3 higher32
ESM-2 higherTieESM3 higher
Compared
ESM-2, ESM3

3 follow-up actions, latest 18 November 2026. Next check 2 December 2026

ESM3 relative to ESM-2: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design is outside the range of other studies

  • Low severity
  • Not yet reviewed
  • Gap outlier
  • Same task, different protocols

On ProteinGym mutation effects, ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design reports ESM3 at a median relative difference of +0.20 from ESM-2; other studies report -0.06 to -0.01.

In 2 papers, ESM3 scores between 1% and 6% below ESM-2. In ProteinGym, ESM3 scores 20% above ESM-2.

How far ESM3 scores from ESM-2 in each paperThe other papers range from −6% to −1%. ProteinGym (disagrees): +20%. HyCell CITATION.cff: −1%. pfmbench primary benchmark evidence: −6%.ProteinGym: +20%ProteinGym +20%HyCell CITATION.cff: −1%pfmbench primary benchmark evidence: −6%−6%= ESM-2+20%
Range of the other papers
Compared
ESM3, ESM-2

Not reviewed yet.

LRT and MetaLR: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks LRT above MetaLR in 15 of 17 comparisons; 2 of 2 other studies rank MetaLR above LRT.

2 papers rank MetaLR above LRT in 12 of their 13 comparisons and LRT higher in 1. Comprehensive assessment… ranks LRT above MetaLR in 15 of its 20 comparisons and MetaLR higher in 2; 3 are ties.

Comparisons in which each paper scores MetaLR or LRT higher2 other papers: MetaLR higher in 12, LRT higher in 1, 0 ties. Comprehensive assessment… (disagrees): MetaLR higher in 2, LRT higher in 15, 3 ties.2 other papers12 MetaLR higher121 LRT higherDisagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations2 MetaLR higher23 tie315 LRT higher15
MetaLR higherTieLRT higher
Compared
LRT, MetaLR

Not reviewed yet.

CADD and MetaLR: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above CADD in 6 of 7 comparisons; 2 of 2 other studies rank CADD above MetaLR.

3 papers rank CADD above MetaLR in 19 of their 21 comparisons and MetaLR higher in 2. Livesey and Marsh 2025 ranks MetaLR above CADD in 6 of its 7 comparisons and CADD higher in 1.

Comparisons in which each paper scores CADD or MetaLR higher3 other papers: CADD higher in 19, MetaLR higher in 2, 0 ties. Livesey and Marsh 2025 (disagrees): CADD higher in 1, MetaLR higher in 6, 0 ties.3 other papers19 CADD higher192 MetaLR higher2Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 CADD higher16 MetaLR higher6
CADD higherTieMetaLR higher
Compared
CADD, MetaLR

Not reviewed yet.

CADD and FATHMM-XF: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks FATHMM-XF above CADD in 10 of 10 comparisons; 2 of 2 other studies rank CADD above FATHMM-XF.

2 papers rank CADD above FATHMM-XF in all 12 comparisons that are not ties (1 is a tie). Comprehensive assessment… ranks FATHMM-XF above CADD in all 10 of its comparisons.

Comparisons in which each paper scores CADD or FATHMM-XF higher2 other papers: CADD higher in 12, FATHMM-XF higher in 0, 1 tie. Comprehensive assessment… (disagrees): CADD higher in 0, FATHMM-XF higher in 10, 0 ties.2 other papers12 CADD higher121 tieDisagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations10 FATHMM-XF higher10
CADD higherTieFATHMM-XF higher
Compared
CADD, FATHMM-XF

Not reviewed yet.

MVP and ESM-1b: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks MVP above ESM-1b in 2 of 2 comparisons; 2 of 2 other studies rank ESM-1b above MVP.

2 papers rank ESM-1b above MVP in all 13 of their comparisons. Brandes et al. 2023 ranks MVP above ESM-1b in all 2 comparisons that are not ties (2 are ties).

Comparisons in which each paper scores ESM-1b or MVP higher2 other papers: ESM-1b higher in 13, MVP higher in 0, 0 ties. Brandes et al. 2023 (disagrees): ESM-1b higher in 0, MVP higher in 2, 2 ties.2 other papers13 ESM-1b higher13Disagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods2 tie22 MVP higher2
ESM-1b higherTieMVP higher
Compared
MVP, ESM-1b

Not reviewed yet.

RNA-FM and RNAErnie: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks RNA-FM above RNAErnie in 28 of 36 comparisons; 2 of 2 other studies rank RNAErnie above RNA-FM.

2 papers rank RNAErnie above RNA-FM in all 6 comparisons that are not ties (13 are ties). Zero-shot benchmarking of RNA language… ranks RNA-FM above RNAErnie in 28 of its 36 comparisons and RNAErnie higher in 8.

Comparisons in which each paper scores RNAErnie or RNA-FM higher2 other papers: RNAErnie higher in 6, RNA-FM higher in 0, 13 ties. Zero-shot benchmarking of RNA language… (disagrees): RNAErnie higher in 8, RNA-FM higher in 28, 0 ties.2 other papers6 RNAErnie higher613 tie13Disagrees: Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning8 RNAErnie higher828 RNA-FM higher28
RNAErnie higherTieRNA-FM higher
Compared
RNA-FM, RNAErnie

Not reviewed yet.

DANN and GERP: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks GERP above DANN in 5 of 6 comparisons; 2 of 2 other studies rank DANN above GERP.

2 papers rank DANN above GERP in all 12 comparisons that are not ties (1 is a tie). Clinical likelihood ratios and balanced… ranks GERP above DANN in 5 of its 6 comparisons and DANN higher in 1.

Comparisons in which each paper scores DANN or GERP higher2 other papers: DANN higher in 12, GERP higher in 0, 1 tie. Clinical likelihood ratios and balanced… (disagrees): DANN higher in 1, GERP higher in 5, 0 ties.2 other papers12 DANN higher121 tieDisagrees: Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes1 DANN higher15 GERP higher5
DANN higherTieGERP higher
Compared
DANN, GERP

Not reviewed yet.

ClinPred and REVEL: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks REVEL above ClinPred in 6 of 7 comparisons; 2 of 2 other studies rank ClinPred above REVEL.

3 papers rank ClinPred above REVEL in 10 of their 12 comparisons and REVEL higher in 1; 1 is a tie. Livesey and Marsh 2025 ranks REVEL above ClinPred in 6 of its 7 comparisons and ClinPred higher in 1.

Comparisons in which each paper scores ClinPred or REVEL higher3 other papers: ClinPred higher in 10, REVEL higher in 1, 1 tie. Livesey and Marsh 2025 (disagrees): ClinPred higher in 1, REVEL higher in 6, 0 ties.3 other papers10 ClinPred higher101 tie1 REVEL higherDisagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 ClinPred higher16 REVEL higher6
ClinPred higherTieREVEL higher
Compared
ClinPred, REVEL

Not reviewed yet.

BayesDel and ESM-1b: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above BayesDel in 6 of 7 comparisons; 2 of 2 other studies rank BayesDel above ESM-1b.

2 papers rank BayesDel above ESM-1b in all 10 of their comparisons. Livesey and Marsh 2025 ranks ESM-1b above BayesDel in 6 of its 7 comparisons and BayesDel higher in 1.

Comparisons in which each paper scores BayesDel or ESM-1b higher2 other papers: BayesDel higher in 10, ESM-1b higher in 0, 0 ties. Livesey and Marsh 2025 (disagrees): BayesDel higher in 1, ESM-1b higher in 6, 0 ties.2 other papers10 BayesDel higher10Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 BayesDel higher16 ESM-1b higher6
BayesDel higherTieESM-1b higher
Compared
BayesDel, ESM-1b

Not reviewed yet.

MVP and VEST4: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MVP above VEST4 in 3 of 4 comparisons; 2 of 2 other studies rank VEST4 above MVP.

3 papers rank VEST4 above MVP in 17 of their 23 comparisons and MVP higher in 6. Clinical likelihood ratios and balanced… ranks MVP above VEST4 in 3 of its 5 comparisons and VEST4 higher in 1; 1 is a tie.

Comparisons in which each paper scores VEST4 or MVP higher3 other papers: VEST4 higher in 17, MVP higher in 6, 0 ties. Clinical likelihood ratios and balanced… (disagrees): VEST4 higher in 1, MVP higher in 3, 1 tie.3 other papers17 VEST4 higher176 MVP higher6Disagrees: Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes1 VEST4 higher11 tie13 MVP higher3
VEST4 higherTieMVP higher
Compared
MVP, VEST4

Not reviewed yet.

Caduceus and GENA-LM: GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models ranks GENA-LM above Caduceus in 41 of 48 comparisons; 2 of 2 other studies rank Caduceus above GENA-LM.

3 papers rank Caduceus above GENA-LM in 23 of their 55 comparisons and GENA-LM higher in 31; 1 is a tie. Of those papers, GENEB leaderboard ranks GENA-LM higher on its own (30 of 45). GenBench ranks GENA-LM above Caduceus in 41 of its 51 comparisons and Caduceus higher in 7; 3 are ties.

Comparisons in which each paper scores Caduceus or GENA-LM higher3 other papers: Caduceus higher in 23, GENA-LM higher in 31, 1 tie. GenBench (disagrees): Caduceus higher in 7, GENA-LM higher in 41, 3 ties.3 other papers23 Caduceus higher231 tie31 GENA-LM higher31Disagrees: GenBenchGenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models7 Caduceus higher73 tie41 GENA-LM higher41
Caduceus higherTieGENA-LM higher
Compared
Caduceus, GENA-LM

Not reviewed yet.

CADD and MVP: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks MVP above CADD in 10 of 10 comparisons; 2 of 2 other studies rank CADD above MVP.

3 papers rank CADD above MVP in 11 of their 18 comparisons and MVP higher in 4; 3 are ties. Of those papers, Clinical likelihood ratios and balanced… ranks MVP higher on its own (2 of 5). Comprehensive assessment… ranks MVP above CADD in all 10 of its comparisons.

Comparisons in which each paper scores CADD or MVP higher3 other papers: CADD higher in 11, MVP higher in 4, 3 ties. Comprehensive assessment… (disagrees): CADD higher in 0, MVP higher in 10, 0 ties.3 other papers11 CADD higher113 tie34 MVP higher4Disagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations10 MVP higher10
CADD higherTieMVP higher
Compared
CADD, MVP

Not reviewed yet.

DNABERT-2 and Nucleotide Transformer: GenomeOcean: An Efficient Genome Foundation Model Trained on Large-Scale Metagenomic Assemblies reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

GenomeOcean: An Efficient Genome Foundation Model Trained on Large-Scale Metagenomic Assemblies ranks DNABERT-2 above Nucleotide Transformer in 3 of 3 comparisons; 7 of 8 other studies rank Nucleotide Transformer above DNABERT-2.

10 papers rank Nucleotide Transformer above DNABERT-2 in 104 of their 171 comparisons and DNABERT-2 higher in 52; 15 are ties. Of those papers, DNABERT-2, BEND rank DNABERT-2 higher on their own (24 of 28, 5 of 7). GenomeOcean 2025 ranks DNABERT-2 above Nucleotide Transformer in all 3 of its comparisons.

Comparisons in which each paper scores Nucleotide Transformer or DNABERT-2 higher10 other papers: Nucleotide Transformer higher in 104, DNABERT-2 higher in 52, 15 ties. GenomeOcean 2025 (disagrees): Nucleotide Transformer higher in 0, DNABERT-2 higher in 3, 0 ties.10 other papers104 Nucleotide Transformer higher10415 tie52 DNABERT-2 higher52Disagrees: GenomeOcean 2025GenomeOcean: An Efficient Genome Foundation Model Trained on Large-Scale Metagenomic Assemblies3 DNABERT-2 higher3
Nucleotide Transformer higherTieDNABERT-2 higher
Compared
DNABERT-2, Nucleotide Transformer

Not reviewed yet.

Geneformer and scFoundation: Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines ranks Geneformer above scFoundation in 2 of 2 comparisons; 2 of 2 other studies rank scFoundation above Geneformer.

2 papers rank scFoundation above Geneformer in 13 of their 16 comparisons and Geneformer higher in 3. Deep-learning-based gene perturbation… ranks Geneformer above scFoundation in all 2 of its comparisons.

Comparisons in which each paper scores scFoundation or Geneformer higher2 other papers: scFoundation higher in 13, Geneformer higher in 3, 0 ties. Deep-learning-based gene perturbation… (disagrees): scFoundation higher in 0, Geneformer higher in 2, 0 ties.2 other papers13 scFoundation higher133 Geneformer higher3Disagrees: Deep-learning-based gene perturbation…Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines2 Geneformer higher2
scFoundation higherTieGeneformer higher
Compared
Geneformer, scFoundation

Not reviewed yet.

MVP and REVEL: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MVP above REVEL in 4 of 5 comparisons; 3 of 3 other studies rank REVEL above MVP.

3 papers rank REVEL above MVP in all 19 comparisons that are not ties (4 are ties). Clinical likelihood ratios and balanced… ranks MVP above REVEL in 4 of its 5 comparisons and REVEL higher in 1.

Comparisons in which each paper scores REVEL or MVP higher3 other papers: REVEL higher in 19, MVP higher in 0, 4 ties. Clinical likelihood ratios and balanced… (disagrees): REVEL higher in 1, MVP higher in 4, 0 ties.3 other papers19 REVEL higher194 tie4Disagrees: Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes1 REVEL higher14 MVP higher4
REVEL higherTieMVP higher
Compared
MVP, REVEL

Not reviewed yet.

M-CAP and PrimateAI: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks M-CAP above PrimateAI in 7 of 7 comparisons; 2 of 2 other studies rank PrimateAI above M-CAP.

3 papers rank PrimateAI above M-CAP in 24 of their 30 comparisons and M-CAP higher in 6. Livesey and Marsh 2025 ranks M-CAP above PrimateAI in all 7 of its comparisons.

Comparisons in which each paper scores PrimateAI or M-CAP higher3 other papers: PrimateAI higher in 24, M-CAP higher in 6, 0 ties. Livesey and Marsh 2025 (disagrees): PrimateAI higher in 0, M-CAP higher in 7, 0 ties.3 other papers24 PrimateAI higher246 M-CAP higher6Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 M-CAP higher7
PrimateAI higherTieM-CAP higher
Compared
M-CAP, PrimateAI

Not reviewed yet.

FATHMM-MKL and MutationAssessor: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above FATHMM-MKL in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-MKL above MutationAssessor.

2 papers rank FATHMM-MKL above MutationAssessor in 19 of their 25 comparisons and MutationAssessor higher in 6. Livesey and Marsh 2025 ranks MutationAssessor above FATHMM-MKL in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-MKL or MutationAssessor higher2 other papers: FATHMM-MKL higher in 19, MutationAssessor higher in 6, 0 ties. Livesey and Marsh 2025 (disagrees): FATHMM-MKL higher in 0, MutationAssessor higher in 7, 0 ties.2 other papers19 FATHMM-MKL higher196 MutationAssessor higher6Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MutationAssessor higher7
FATHMM-MKL higherTieMutationAssessor higher
Compared
FATHMM-MKL, MutationAssessor

Not reviewed yet.

GenoCanyon and SIFT: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks GenoCanyon above SIFT in 6 of 8 comparisons; 3 of 3 other studies rank SIFT above GenoCanyon.

3 papers rank SIFT above GenoCanyon in 14 of their 16 comparisons and GenoCanyon higher in 1; 1 is a tie. Comprehensive assessment… ranks GenoCanyon above SIFT in 6 of its 10 comparisons and SIFT higher in 2; 2 are ties.

Comparisons in which each paper scores SIFT or GenoCanyon higher3 other papers: SIFT higher in 14, GenoCanyon higher in 1, 1 tie. Comprehensive assessment… (disagrees): SIFT higher in 2, GenoCanyon higher in 6, 2 ties.3 other papers14 SIFT higher141 tie1 GenoCanyon higherDisagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations2 SIFT higher22 tie26 GenoCanyon higher6
SIFT higherTieGenoCanyon higher
Compared
GenoCanyon, SIFT

Not reviewed yet.

DNABERT-2 and Nucleotide Transformer: DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genomes reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genomes ranks DNABERT-2 above Nucleotide Transformer in 24 of 26 comparisons; 7 of 8 other studies rank Nucleotide Transformer above DNABERT-2.

10 papers rank Nucleotide Transformer above DNABERT-2 in 102 of their 146 comparisons and DNABERT-2 higher in 31; 13 are ties. Of those papers, BEND, GenomeOcean 2025 rank DNABERT-2 higher on their own (5 of 7, 3 of 3). DNABERT-2 ranks DNABERT-2 above Nucleotide Transformer in 24 of its 28 comparisons and Nucleotide Transformer higher in 2; 2 are ties.

Comparisons in which each paper scores Nucleotide Transformer or DNABERT-2 higher10 other papers: Nucleotide Transformer higher in 102, DNABERT-2 higher in 31, 13 ties. DNABERT-2 (disagrees): Nucleotide Transformer higher in 2, DNABERT-2 higher in 24, 2 ties.10 other papers102 Nucleotide Transformer higher10213 tie31 DNABERT-2 higher31Disagrees: DNABERT-2DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genomes2 Nucleotide Transformer higher2 tie24 DNABERT-2 higher24
Nucleotide Transformer higherTieDNABERT-2 higher
Compared
DNABERT-2, Nucleotide Transformer

Not reviewed yet.

MetaLR and ESM-1b: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks MetaLR above ESM-1b in 2 of 2 comparisons; 2 of 2 other studies rank ESM-1b above MetaLR.

2 papers rank ESM-1b above MetaLR in all 13 of their comparisons. Brandes et al. 2023 ranks MetaLR above ESM-1b in all 2 comparisons that are not ties (2 are ties).

Comparisons in which each paper scores ESM-1b or MetaLR higher2 other papers: ESM-1b higher in 13, MetaLR higher in 0, 0 ties. Brandes et al. 2023 (disagrees): ESM-1b higher in 0, MetaLR higher in 2, 2 ties.2 other papers13 ESM-1b higher13Disagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods2 tie22 MetaLR higher2
ESM-1b higherTieMetaLR higher
Compared
MetaLR, ESM-1b

Not reviewed yet.

MetaSVM and ESM-1b: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks MetaSVM above ESM-1b in 2 of 2 comparisons; 2 of 2 other studies rank ESM-1b above MetaSVM.

2 papers rank ESM-1b above MetaSVM in all 13 of their comparisons. Brandes et al. 2023 ranks MetaSVM above ESM-1b in all 2 comparisons that are not ties (2 are ties).

Comparisons in which each paper scores ESM-1b or MetaSVM higher2 other papers: ESM-1b higher in 13, MetaSVM higher in 0, 0 ties. Brandes et al. 2023 (disagrees): ESM-1b higher in 0, MetaSVM higher in 2, 2 ties.2 other papers13 ESM-1b higher13Disagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods2 tie22 MetaSVM higher2
ESM-1b higherTieMetaSVM higher
Compared
MetaSVM, ESM-1b

Not reviewed yet.

FATHMM-XF and M-CAP: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks M-CAP above FATHMM-XF in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above M-CAP.

2 papers rank FATHMM-XF above M-CAP in 22 of their 26 comparisons and M-CAP higher in 3; 1 is a tie. Livesey and Marsh 2025 ranks M-CAP above FATHMM-XF in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or M-CAP higher2 other papers: FATHMM-XF higher in 22, M-CAP higher in 3, 1 tie. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, M-CAP higher in 7, 0 ties.2 other papers22 FATHMM-XF higher221 tie3 M-CAP higher3Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 M-CAP higher7
FATHMM-XF higherTieM-CAP higher
Compared
FATHMM-XF, M-CAP

Not reviewed yet.

FATHMM-MKL and MetaLR: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above FATHMM-MKL in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-MKL above MetaLR.

2 papers rank FATHMM-MKL above MetaLR in 18 of their 25 comparisons and MetaLR higher in 5; 2 are ties. Livesey and Marsh 2025 ranks MetaLR above FATHMM-MKL in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-MKL or MetaLR higher2 other papers: FATHMM-MKL higher in 18, MetaLR higher in 5, 2 ties. Livesey and Marsh 2025 (disagrees): FATHMM-MKL higher in 0, MetaLR higher in 7, 0 ties.2 other papers18 FATHMM-MKL higher182 tie5 MetaLR higher5Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MetaLR higher7
FATHMM-MKL higherTieMetaLR higher
Compared
FATHMM-MKL, MetaLR

Not reviewed yet.

phyloP and SIFT: An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks phyloP above SIFT in 3 of 4 comparisons; 2 of 2 other studies rank SIFT above phyloP.

3 papers rank SIFT above phyloP in 14 of their 16 comparisons and phyloP higher in 2. An openly licensed benchmark… ranks phyloP above SIFT in 3 of its 6 comparisons and SIFT higher in 1; 2 are ties.

Comparisons in which each paper scores SIFT or phyloP higher3 other papers: SIFT higher in 14, phyloP higher in 2, 0 ties. An openly licensed benchmark… (disagrees): SIFT higher in 1, phyloP higher in 3, 2 ties.3 other papers14 SIFT higher142 phyloP higher2Disagrees: An openly licensed benchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers1 SIFT higher12 tie23 phyloP higher3
SIFT higherTiephyloP higher
Compared
phyloP, SIFT

Not reviewed yet.

CADD and PrimateAI: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks PrimateAI above CADD in 10 of 10 comparisons; 2 of 2 other studies rank CADD above PrimateAI.

3 papers rank CADD above PrimateAI in 15 of their 18 comparisons and PrimateAI higher in 2; 1 is a tie. Comprehensive assessment… ranks PrimateAI above CADD in all 10 of its comparisons.

Comparisons in which each paper scores CADD or PrimateAI higher3 other papers: CADD higher in 15, PrimateAI higher in 2, 1 tie. Comprehensive assessment… (disagrees): CADD higher in 0, PrimateAI higher in 10, 0 ties.3 other papers15 CADD higher151 tie2 PrimateAI higher2Disagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations10 PrimateAI higher10
CADD higherTiePrimateAI higher
Compared
CADD, PrimateAI

Not reviewed yet.

FATHMM-XF and MetaLR: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above FATHMM-XF in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above MetaLR.

2 papers rank FATHMM-XF above MetaLR in 22 of their 26 comparisons and MetaLR higher in 3; 1 is a tie. Livesey and Marsh 2025 ranks MetaLR above FATHMM-XF in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or MetaLR higher2 other papers: FATHMM-XF higher in 22, MetaLR higher in 3, 1 tie. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, MetaLR higher in 7, 0 ties.2 other papers22 FATHMM-XF higher221 tie3 MetaLR higher3Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MetaLR higher7
FATHMM-XF higherTieMetaLR higher
Compared
FATHMM-XF, MetaLR

Not reviewed yet.

MetaSVM and MutationAssessor: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MetaSVM above MutationAssessor in 4 of 5 comparisons; 2 of 2 other studies rank MutationAssessor above MetaSVM.

3 papers rank MutationAssessor above MetaSVM in 22 of their 33 comparisons and MetaSVM higher in 10; 1 is a tie. Clinical likelihood ratios and balanced… ranks MetaSVM above MutationAssessor in 4 of its 6 comparisons and MutationAssessor higher in 1; 1 is a tie.

Comparisons in which each paper scores MutationAssessor or MetaSVM higher3 other papers: MutationAssessor higher in 22, MetaSVM higher in 10, 1 tie. Clinical likelihood ratios and balanced… (disagrees): MutationAssessor higher in 1, MetaSVM higher in 4, 1 tie.3 other papers22 MutationAssessor higher221 tie10 MetaSVM higher10Disagrees: Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes1 MutationAssessor higher11 tie14 MetaSVM higher4
MutationAssessor higherTieMetaSVM higher
Compared
MetaSVM, MutationAssessor

Not reviewed yet.

MutPred and PrimateAI: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks PrimateAI above MutPred in 9 of 10 comparisons; 2 of 2 other studies rank MutPred above PrimateAI.

4 papers rank MutPred above PrimateAI in 17 of their 20 comparisons and PrimateAI higher in 3. Comprehensive assessment… ranks PrimateAI above MutPred in 9 of its 10 comparisons and MutPred higher in 1.

Comparisons in which each paper scores MutPred or PrimateAI higher4 other papers: MutPred higher in 17, PrimateAI higher in 3, 0 ties. Comprehensive assessment… (disagrees): MutPred higher in 1, PrimateAI higher in 9, 0 ties.4 other papers17 MutPred higher173 PrimateAI higher3Disagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations1 MutPred higher19 PrimateAI higher9
MutPred higherTiePrimateAI higher
Compared
MutPred, PrimateAI

Not reviewed yet.

FATHMM-XF and MetaSVM: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaSVM above FATHMM-XF in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above MetaSVM.

2 papers rank FATHMM-XF above MetaSVM in 22 of their 26 comparisons and MetaSVM higher in 4. Livesey and Marsh 2025 ranks MetaSVM above FATHMM-XF in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or MetaSVM higher2 other papers: FATHMM-XF higher in 22, MetaSVM higher in 4, 0 ties. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, MetaSVM higher in 7, 0 ties.2 other papers22 FATHMM-XF higher224 MetaSVM higher4Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MetaSVM higher7
FATHMM-XF higherTieMetaSVM higher
Compared
FATHMM-XF, MetaSVM

Not reviewed yet.

Pangolin and SpliceAI: Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) ranks SpliceAI above Pangolin in 27 of 28 comparisons; 3 of 3 other studies rank Pangolin above SpliceAI.

7 papers rank Pangolin above SpliceAI in 29 of their 72 comparisons and SpliceAI higher in 18; 25 are ties. Genome modelling and design across all… ranks SpliceAI above Pangolin in 27 of its 28 comparisons and Pangolin higher in 1.

Comparisons in which each paper scores Pangolin or SpliceAI higher7 other papers: Pangolin higher in 29, SpliceAI higher in 18, 25 ties. Genome modelling and design across all… (disagrees): Pangolin higher in 1, SpliceAI higher in 27, 0 ties.7 other papers29 Pangolin higher2925 tie2518 SpliceAI higher18Disagrees: Genome modelling and design across all…Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026)1 Pangolin higher27 SpliceAI higher27
Pangolin higherTieSpliceAI higher
Compared
Pangolin, SpliceAI

Not reviewed yet.

PrimateAI and SIFT: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks SIFT above PrimateAI in 6 of 6 comparisons; 2 of 2 other studies rank PrimateAI above SIFT.

4 papers rank PrimateAI above SIFT in 25 of their 32 comparisons and SIFT higher in 7. Of those papers, ProteinGym ranks SIFT higher on its own (1 of 1). Livesey and Marsh 2025 ranks SIFT above PrimateAI in all 6 comparisons that are not ties (1 is a tie).

Comparisons in which each paper scores PrimateAI or SIFT higher4 other papers: PrimateAI higher in 25, SIFT higher in 7, 0 ties. Livesey and Marsh 2025 (disagrees): PrimateAI higher in 0, SIFT higher in 6, 1 tie.4 other papers25 PrimateAI higher257 SIFT higher7Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 tie16 SIFT higher6
PrimateAI higherTieSIFT higher
Compared
PrimateAI, SIFT

Not reviewed yet.

REVEL and BayesDel: An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks BayesDel above REVEL in 6 of 6 comparisons; 2 of 2 other studies rank REVEL above BayesDel.

3 papers rank REVEL above BayesDel in 11 of their 14 comparisons and BayesDel higher in 2; 1 is a tie. Of those papers, ProteinGym ranks BayesDel higher on its own (1 of 1). An openly licensed benchmark… ranks BayesDel above REVEL in all 6 of its comparisons.

Comparisons in which each paper scores REVEL or BayesDel higher3 other papers: REVEL higher in 11, BayesDel higher in 2, 1 tie. An openly licensed benchmark… (disagrees): REVEL higher in 0, BayesDel higher in 6, 0 ties.3 other papers11 REVEL higher111 tie2 BayesDel higher2Disagrees: An openly licensed benchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers6 BayesDel higher6
REVEL higherTieBayesDel higher
Compared
REVEL, BayesDel

Not reviewed yet.

LRT and MetaSVM: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks LRT above MetaSVM in 16 of 20 comparisons; 2 of 2 other studies rank MetaSVM above LRT.

2 papers rank MetaSVM above LRT in 12 of their 13 comparisons and LRT higher in 1. Comprehensive assessment… ranks LRT above MetaSVM in 16 of its 20 comparisons and MetaSVM higher in 4.

Comparisons in which each paper scores MetaSVM or LRT higher2 other papers: MetaSVM higher in 12, LRT higher in 1, 0 ties. Comprehensive assessment… (disagrees): MetaSVM higher in 4, LRT higher in 16, 0 ties.2 other papers12 MetaSVM higher121 LRT higherDisagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations4 MetaSVM higher416 LRT higher16
MetaSVM higherTieLRT higher
Compared
LRT, MetaSVM

Not reviewed yet.

Nucleotide Transformer v2 and Nucleotide Transformer: GENEB leaderboard (leaderboard.json, 13 task-type aggregates) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

GENEB leaderboard (leaderboard.json, 13 task-type aggregates) ranks Nucleotide Transformer above Nucleotide Transformer v2 in 37 of 44 comparisons; 2 of 2 other studies rank Nucleotide Transformer v2 above Nucleotide Transformer.

2 papers rank Nucleotide Transformer v2 above Nucleotide Transformer in 33 of their 46 comparisons and Nucleotide Transformer higher in 7; 6 are ties. GENEB leaderboard ranks Nucleotide Transformer above Nucleotide Transformer v2 in 37 of its 45 comparisons and Nucleotide Transformer v2 higher in 7; 1 is a tie.

Comparisons in which each paper scores Nucleotide Transformer v2 or Nucleotide Transformer higher2 other papers: Nucleotide Transformer v2 higher in 33, Nucleotide Transformer higher in 7, 6 ties. GENEB leaderboard (disagrees): Nucleotide Transformer v2 higher in 7, Nucleotide Transformer higher in 37, 1 tie.2 other papers33 Nucleotide Transformer v2 higher336 tie67 Nucleotide Transformer higher7Disagrees: GENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)7 Nucleotide Transformer v2 higher71 tie37 Nucleotide Transformer higher37
Nucleotide Transformer v2 higherTieNucleotide Transformer higher
Compared
Nucleotide Transformer v2, Nucleotide Transformer

Not reviewed yet.

MutPred and ESM-1b: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above MutPred in 6 of 7 comparisons; 2 of 2 other studies rank MutPred above ESM-1b.

3 papers rank MutPred above ESM-1b in 8 of their 11 comparisons and ESM-1b higher in 1; 2 are ties. Of those papers, ProteinGym ranks ESM-1b higher on its own (1 of 1). Livesey and Marsh 2025 ranks ESM-1b above MutPred in 6 of its 7 comparisons and MutPred higher in 1.

Comparisons in which each paper scores MutPred or ESM-1b higher3 other papers: MutPred higher in 8, ESM-1b higher in 1, 2 ties. Livesey and Marsh 2025 (disagrees): MutPred higher in 1, ESM-1b higher in 6, 0 ties.3 other papers8 MutPred higher82 tie21 ESM-1b higher1Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 MutPred higher16 ESM-1b higher6
MutPred higherTieESM-1b higher
Compared
MutPred, ESM-1b

Not reviewed yet.

MPC and MutationAssessor: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above MPC in 6 of 7 comparisons; 2 of 2 other studies rank MPC above MutationAssessor.

2 papers rank MPC above MutationAssessor in all 16 of their comparisons. Livesey and Marsh 2025 ranks MutationAssessor above MPC in 6 of its 7 comparisons and MPC higher in 1.

Comparisons in which each paper scores MPC or MutationAssessor higher2 other papers: MPC higher in 16, MutationAssessor higher in 0, 0 ties. Livesey and Marsh 2025 (disagrees): MPC higher in 1, MutationAssessor higher in 6, 0 ties.2 other papers16 MPC higher16Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 MPC higher16 MutationAssessor higher6
MPC higherTieMutationAssessor higher
Compared
MPC, MutationAssessor

Not reviewed yet.

MetaSVM and PrimateAI: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaSVM above PrimateAI in 7 of 7 comparisons; 2 of 2 other studies rank PrimateAI above MetaSVM.

3 papers rank PrimateAI above MetaSVM in 25 of their 32 comparisons and MetaSVM higher in 5; 2 are ties. Of those papers, Clinical likelihood ratios and balanced… ranks MetaSVM higher on its own (4 of 6). Livesey and Marsh 2025 ranks MetaSVM above PrimateAI in all 7 of its comparisons.

Comparisons in which each paper scores PrimateAI or MetaSVM higher3 other papers: PrimateAI higher in 25, MetaSVM higher in 5, 2 ties. Livesey and Marsh 2025 (disagrees): PrimateAI higher in 0, MetaSVM higher in 7, 0 ties.3 other papers25 PrimateAI higher252 tie5 MetaSVM higher5Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MetaSVM higher7
PrimateAI higherTieMetaSVM higher
Compared
MetaSVM, PrimateAI

Not reviewed yet.

Caduceus and HyenaDNA: A Phylogenetic Approach to Genomic Language Modeling reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

A Phylogenetic Approach to Genomic Language Modeling ranks Caduceus above HyenaDNA in 9 of 12 comparisons; 2 of 2 other studies rank HyenaDNA above Caduceus.

5 papers rank HyenaDNA above Caduceus in 63 of their 133 comparisons and Caduceus higher in 49; 21 are ties. Of those papers, DNALONGBENCH ranks Caduceus higher on its own (16 of 27). A Phylogenetic Approach to Genomic… ranks Caduceus above HyenaDNA in 9 of its 12 comparisons and HyenaDNA higher in 3.

Comparisons in which each paper scores HyenaDNA or Caduceus higher5 other papers: HyenaDNA higher in 63, Caduceus higher in 49, 21 ties. A Phylogenetic Approach to Genomic… (disagrees): HyenaDNA higher in 3, Caduceus higher in 9, 0 ties.5 other papers63 HyenaDNA higher6321 tie2149 Caduceus higher49Disagrees: A Phylogenetic Approach to Genomic…A Phylogenetic Approach to Genomic Language Modeling3 HyenaDNA higher39 Caduceus higher9
HyenaDNA higherTieCaduceus higher
Compared
Caduceus, HyenaDNA

Not reviewed yet.

LRT and SIFT: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks LRT above SIFT in 18 of 20 comparisons; 2 of 2 other studies rank SIFT above LRT.

4 papers rank SIFT above LRT in 14 of their 17 comparisons and LRT higher in 2; 1 is a tie. Comprehensive assessment… ranks LRT above SIFT in 18 of its 20 comparisons and SIFT higher in 2.

Comparisons in which each paper scores SIFT or LRT higher4 other papers: SIFT higher in 14, LRT higher in 2, 1 tie. Comprehensive assessment… (disagrees): SIFT higher in 2, LRT higher in 18, 0 ties.4 other papers14 SIFT higher141 tie2 LRT higher2Disagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations2 SIFT higher218 LRT higher18
SIFT higherTieLRT higher
Compared
LRT, SIFT

Not reviewed yet.

MVP and PolyPhen-2: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks PolyPhen-2 above MVP in 6 of 6 comparisons; 3 of 3 other studies rank MVP above PolyPhen-2.

3 papers rank MVP above PolyPhen-2 in 18 of their 21 comparisons and PolyPhen-2 higher in 2; 1 is a tie. Livesey and Marsh 2025 ranks PolyPhen-2 above MVP in all 6 of its comparisons.

Comparisons in which each paper scores MVP or PolyPhen-2 higher3 other papers: MVP higher in 18, PolyPhen-2 higher in 2, 1 tie. Livesey and Marsh 2025 (disagrees): MVP higher in 0, PolyPhen-2 higher in 6, 0 ties.3 other papers18 MVP higher181 tie2 PolyPhen-2 higher2Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)6 PolyPhen-2 higher6
MVP higherTiePolyPhen-2 higher
Compared
MVP, PolyPhen-2

Not reviewed yet.

DANN and MetaLR: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks DANN above MetaLR in 10 of 10 comparisons; 2 of 2 other studies rank MetaLR above DANN.

3 papers rank MetaLR above DANN in 15 of their 19 comparisons and DANN higher in 3; 1 is a tie. Comprehensive assessment… ranks DANN above MetaLR in all 10 of its comparisons.

Comparisons in which each paper scores MetaLR or DANN higher3 other papers: MetaLR higher in 15, DANN higher in 3, 1 tie. Comprehensive assessment… (disagrees): MetaLR higher in 0, DANN higher in 10, 0 ties.3 other papers15 MetaLR higher151 tie3 DANN higher3Disagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations10 DANN higher10
MetaLR higherTieDANN higher
Compared
DANN, MetaLR

Not reviewed yet.

Nucleotide Transformer and RNA-FM: NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction ranks RNA-FM above Nucleotide Transformer in 22 of 25 comparisons; 2 of 2 other studies rank Nucleotide Transformer above RNA-FM.

4 papers rank Nucleotide Transformer above RNA-FM in 42 of their 81 comparisons and RNA-FM higher in 25; 14 are ties. NABench ranks RNA-FM above Nucleotide Transformer in 22 of its 27 comparisons and Nucleotide Transformer higher in 3; 2 are ties.

Comparisons in which each paper scores Nucleotide Transformer or RNA-FM higher4 other papers: Nucleotide Transformer higher in 42, RNA-FM higher in 25, 14 ties. NABench (disagrees): Nucleotide Transformer higher in 3, RNA-FM higher in 22, 2 ties.4 other papers42 Nucleotide Transformer higher4214 tie1425 RNA-FM higher25Disagrees: NABenchNABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction3 Nucleotide Transformer higher32 tie22 RNA-FM higher22
Nucleotide Transformer higherTieRNA-FM higher
Compared
Nucleotide Transformer, RNA-FM

Not reviewed yet.

Evo 2 and Nucleotide Transformer: BacBench LICENSE file at commit 64b7d45 reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

BacBench LICENSE file at commit 64b7d45 ranks Nucleotide Transformer above Evo 2 in 10 of 10 comparisons; 4 of 4 other studies rank Evo 2 above Nucleotide Transformer.

6 papers rank Evo 2 above Nucleotide Transformer in 127 of their 147 comparisons and Nucleotide Transformer higher in 19; 1 is a tie. BacBench LICENSE file at commit 64b7d45 ranks Nucleotide Transformer above Evo 2 in all 10 of its comparisons.

Comparisons in which each paper scores Evo 2 or Nucleotide Transformer higher6 other papers: Evo 2 higher in 127, Nucleotide Transformer higher in 19, 1 tie. BacBench LICENSE file at commit 64b7d45 (disagrees): Evo 2 higher in 0, Nucleotide Transformer higher in 10, 0 ties.6 other papers127 Evo 2 higher1271 tie19 Nucleotide Transformer higher19Disagrees: BacBench LICENSE file at commit 64b7d45BacBench LICENSE file at commit 64b7d4510 Nucleotide Transformer higher10
Evo 2 higherTieNucleotide Transformer higher
Compared
Evo 2, Nucleotide Transformer

Not reviewed yet.

GenoCanyon and phyloP: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks GenoCanyon above phyloP in 3 of 3 comparisons; 2 of 2 other studies rank phyloP above GenoCanyon.

2 papers rank phyloP above GenoCanyon in 11 of their 13 comparisons and GenoCanyon higher in 1; 1 is a tie. Brandes et al. 2023 ranks GenoCanyon above phyloP in all 3 comparisons that are not ties (1 is a tie).

Comparisons in which each paper scores phyloP or GenoCanyon higher2 other papers: phyloP higher in 11, GenoCanyon higher in 1, 1 tie. Brandes et al. 2023 (disagrees): phyloP higher in 0, GenoCanyon higher in 3, 1 tie.2 other papers11 phyloP higher111 tie1 GenoCanyon higherDisagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods1 tie13 GenoCanyon higher3
phyloP higherTieGenoCanyon higher
Compared
GenoCanyon, phyloP

Not reviewed yet.

Evo 2 and phyloP: TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) ranks phyloP above Evo 2 in 2 of 2 comparisons; 2 of 2 other studies rank Evo 2 above phyloP.

3 papers rank Evo 2 above phyloP in 83 of their 107 comparisons and phyloP higher in 16; 8 are ties. TraitGym official metrics ranks phyloP above Evo 2 in all 2 of its comparisons.

Comparisons in which each paper scores Evo 2 or phyloP higher3 other papers: Evo 2 higher in 83, phyloP higher in 16, 8 ties. TraitGym official metrics (disagrees): Evo 2 higher in 0, phyloP higher in 2, 0 ties.3 other papers83 Evo 2 higher838 tie16 phyloP higher16Disagrees: TraitGym official metricsTraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)2 phyloP higher2
Evo 2 higherTiephyloP higher
Compared
Evo 2, phyloP

Not reviewed yet.

scFoundation and scGPT: Csendes et al. 2025, Supplementary Material 4 (Supplementary Tables 1-3) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Csendes et al. 2025, Supplementary Material 4 (Supplementary Tables 1-3) ranks scGPT above scFoundation in 15 of 16 comparisons; 2 of 2 other studies rank scFoundation above scGPT.

3 papers rank scFoundation above scGPT in 18 of their 25 comparisons and scGPT higher in 5; 2 are ties. Csendes et al. 2025 ranks scGPT above scFoundation in 15 of its 20 comparisons and scFoundation higher in 1; 4 are ties.

Comparisons in which each paper scores scFoundation or scGPT higher3 other papers: scFoundation higher in 18, scGPT higher in 5, 2 ties. Csendes et al. 2025 (disagrees): scFoundation higher in 1, scGPT higher in 15, 4 ties.3 other papers18 scFoundation higher182 tie5 scGPT higher5Disagrees: Csendes et al. 2025Csendes et al. 2025, Supplementary Material 4 (Supplementary Tables 1-3)1 scFoundation higher4 tie415 scGPT higher15
scFoundation higherTiescGPT higher
Compared
scFoundation, scGPT

Not reviewed yet.

VEST4 and ESM-1b: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above VEST4 in 6 of 7 comparisons; 2 of 2 other studies rank VEST4 above ESM-1b.

2 papers rank VEST4 above ESM-1b in 7 of their 10 comparisons and ESM-1b higher in 1; 2 are ties. Livesey and Marsh 2025 ranks ESM-1b above VEST4 in 6 of its 7 comparisons and VEST4 higher in 1.

Comparisons in which each paper scores VEST4 or ESM-1b higher2 other papers: VEST4 higher in 7, ESM-1b higher in 1, 2 ties. Livesey and Marsh 2025 (disagrees): VEST4 higher in 1, ESM-1b higher in 6, 0 ties.2 other papers7 VEST4 higher72 tie21 ESM-1b higher1Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 VEST4 higher16 ESM-1b higher6
VEST4 higherTieESM-1b higher
Compared
VEST4, ESM-1b

Not reviewed yet.

First page · Next page