rewirebio.iobenchmarks
Evidence and investigation

Contested results

Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.

These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.

A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.

How the check works, and how each follow-up is recorded: discrepancies between studies.

In this release

  • 192open
  • 6settled
  • 7resolved
  • 3medium severity
  • 202low severity

Release 2026-11-22-bbed7dce2323 · 205 findings in all · rules version 5

Findings

Status: All · Open · Settled · Resolved

Severity: All · Medium · Low

State: All · Not yet reviewed · Under investigation · Explained · Confirmed error · Not a real disagreement

192 findings match, most severe first.

MutationAssessor and PROVEAN: An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks PROVEAN above MutationAssessor in 6 of 6 comparisons; 2 of 2 other studies rank MutationAssessor above PROVEAN.

5 papers rank MutationAssessor above PROVEAN in 21 of their 38 comparisons and PROVEAN higher in 15; 2 are ties. Of those papers, ProteinGym ranks PROVEAN higher on its own (1 of 1). An openly licensed benchmark… ranks PROVEAN above MutationAssessor in all 6 of its comparisons.

Comparisons in which each paper scores MutationAssessor or PROVEAN higher5 other papers: MutationAssessor higher in 21, PROVEAN higher in 15, 2 ties. An openly licensed benchmark… (disagrees): MutationAssessor higher in 0, PROVEAN higher in 6, 0 ties.5 other papers21 MutationAssessor higher212 tie15 PROVEAN higher15Disagrees: An openly licensed benchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers6 PROVEAN higher6
MutationAssessor higherTiePROVEAN higher
Compared
MutationAssessor, PROVEAN

Not reviewed yet.

FATHMM-XF and MutationAssessor: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above FATHMM-XF in 7 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above MutationAssessor.

2 papers rank FATHMM-XF above MutationAssessor in 21 of their 26 comparisons and MutationAssessor higher in 5. Livesey and Marsh 2025 ranks MutationAssessor above FATHMM-XF in all 7 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or MutationAssessor higher2 other papers: FATHMM-XF higher in 21, MutationAssessor higher in 5, 0 ties. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, MutationAssessor higher in 7, 0 ties.2 other papers21 FATHMM-XF higher215 MutationAssessor higher5Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)7 MutationAssessor higher7
FATHMM-XF higherTieMutationAssessor higher
Compared
FATHMM-XF, MutationAssessor

Not reviewed yet.

DNABERT-2 and GENA-LM: GUANinE v1.1 reveals complementarity of supervised and genomic language models reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks GENA-LM above DNABERT-2 in 6 of 6 comparisons; 2 of 2 other studies rank DNABERT-2 above GENA-LM.

4 papers rank DNABERT-2 above GENA-LM in 62 of their 112 comparisons and GENA-LM higher in 26; 24 are ties. GUANinE v1.1 reveals complementarity… ranks GENA-LM above DNABERT-2 in all 6 of its comparisons.

Comparisons in which each paper scores DNABERT-2 or GENA-LM higher4 other papers: DNABERT-2 higher in 62, GENA-LM higher in 26, 24 ties. GUANinE v1.1 reveals complementarity… (disagrees): DNABERT-2 higher in 0, GENA-LM higher in 6, 0 ties.4 other papers62 DNABERT-2 higher6224 tie2426 GENA-LM higher26Disagrees: GUANinE v1.1 reveals complementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language models6 GENA-LM higher6
DNABERT-2 higherTieGENA-LM higher
Compared
DNABERT-2, GENA-LM

Not reviewed yet.

Evo 2 and Evo: BacBench LICENSE file at commit 64b7d45 reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

BacBench LICENSE file at commit 64b7d45 ranks Evo above Evo 2 in 8 of 10 comparisons; 2 of 2 other studies rank Evo 2 above Evo.

2 papers rank Evo 2 above Evo in all 94 of their comparisons. BacBench LICENSE file at commit 64b7d45 ranks Evo above Evo 2 in 8 of its 10 comparisons and Evo 2 higher in 2.

Comparisons in which each paper scores Evo 2 or Evo higher2 other papers: Evo 2 higher in 94, Evo higher in 0, 0 ties. BacBench LICENSE file at commit 64b7d45 (disagrees): Evo 2 higher in 2, Evo higher in 8, 0 ties.2 other papers94 Evo 2 higher94Disagrees: BacBench LICENSE file at commit 64b7d45BacBench LICENSE file at commit 64b7d452 Evo 2 higher28 Evo higher8
Evo 2 higherTieEvo higher
Compared
Evo 2, Evo

Not reviewed yet.

GenoCanyon and MetaSVM: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks GenoCanyon above MetaSVM in 6 of 8 comparisons; 2 of 2 other studies rank MetaSVM above GenoCanyon.

2 papers rank MetaSVM above GenoCanyon in 12 of their 13 comparisons and GenoCanyon higher in 1. Comprehensive assessment… ranks GenoCanyon above MetaSVM in 6 of its 10 comparisons and MetaSVM higher in 2; 2 are ties.

Comparisons in which each paper scores MetaSVM or GenoCanyon higher2 other papers: MetaSVM higher in 12, GenoCanyon higher in 1, 0 ties. Comprehensive assessment… (disagrees): MetaSVM higher in 2, GenoCanyon higher in 6, 2 ties.2 other papers12 MetaSVM higher121 GenoCanyon higherDisagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations2 MetaSVM higher22 tie26 GenoCanyon higher6
MetaSVM higherTieGenoCanyon higher
Compared
GenoCanyon, MetaSVM

Not reviewed yet.

CADD and PolyPhen-2: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks PolyPhen-2 above CADD in 6 of 6 comparisons; 2 of 2 other studies rank CADD above PolyPhen-2.

4 papers rank CADD above PolyPhen-2 in 19 of their 22 comparisons and PolyPhen-2 higher in 2; 1 is a tie. Livesey and Marsh 2025 ranks PolyPhen-2 above CADD in all 6 of its comparisons.

Comparisons in which each paper scores CADD or PolyPhen-2 higher4 other papers: CADD higher in 19, PolyPhen-2 higher in 2, 1 tie. Livesey and Marsh 2025 (disagrees): CADD higher in 0, PolyPhen-2 higher in 6, 0 ties.4 other papers19 CADD higher191 tie2 PolyPhen-2 higher2Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)6 PolyPhen-2 higher6
CADD higherTiePolyPhen-2 higher
Compared
CADD, PolyPhen-2

Not reviewed yet.

M-CAP and MVP: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks M-CAP above MVP in 6 of 7 comparisons; 2 of 2 other studies rank MVP above M-CAP.

3 papers rank MVP above M-CAP in 16 of their 20 comparisons and M-CAP higher in 1; 3 are ties. Livesey and Marsh 2025 ranks M-CAP above MVP in 6 of its 7 comparisons and MVP higher in 1.

Comparisons in which each paper scores MVP or M-CAP higher3 other papers: MVP higher in 16, M-CAP higher in 1, 3 ties. Livesey and Marsh 2025 (disagrees): MVP higher in 1, M-CAP higher in 6, 0 ties.3 other papers16 MVP higher163 tie31 M-CAP higherDisagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 MVP higher16 M-CAP higher6
MVP higherTieM-CAP higher
Compared
M-CAP, MVP

Not reviewed yet.

MVP and PrimateAI: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks PrimateAI above MVP in 10 of 10 comparisons; 2 of 2 other studies rank MVP above PrimateAI.

3 papers rank MVP above PrimateAI in 13 of their 18 comparisons and PrimateAI higher in 3; 2 are ties. Comprehensive assessment… ranks PrimateAI above MVP in all 10 of its comparisons.

Comparisons in which each paper scores MVP or PrimateAI higher3 other papers: MVP higher in 13, PrimateAI higher in 3, 2 ties. Comprehensive assessment… (disagrees): MVP higher in 0, PrimateAI higher in 10, 0 ties.3 other papers13 MVP higher132 tie23 PrimateAI higher3Disagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations10 PrimateAI higher10
MVP higherTiePrimateAI higher
Compared
MVP, PrimateAI

Not reviewed yet.

PolyPhen-2 and PrimateAI: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks PolyPhen-2 above PrimateAI in 6 of 6 comparisons; 2 of 2 other studies rank PrimateAI above PolyPhen-2.

3 papers rank PrimateAI above PolyPhen-2 in 23 of their 32 comparisons and PolyPhen-2 higher in 8; 1 is a tie. Of those papers, Clinical likelihood ratios and balanced… ranks PolyPhen-2 higher on its own (4 of 6). Livesey and Marsh 2025 ranks PolyPhen-2 above PrimateAI in all 6 of its comparisons.

Comparisons in which each paper scores PrimateAI or PolyPhen-2 higher3 other papers: PrimateAI higher in 23, PolyPhen-2 higher in 8, 1 tie. Livesey and Marsh 2025 (disagrees): PrimateAI higher in 0, PolyPhen-2 higher in 6, 0 ties.3 other papers23 PrimateAI higher231 tie8 PolyPhen-2 higher8Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)6 PolyPhen-2 higher6
PrimateAI higherTiePolyPhen-2 higher
Compared
PolyPhen-2, PrimateAI

Not reviewed yet.

DANN and M-CAP: Comprehensive assessment of computational algorithms in predicting cancer driver mutations reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks DANN above M-CAP in 5 of 5 comparisons; 2 of 2 other studies rank M-CAP above DANN.

3 papers rank M-CAP above DANN in 13 of their 17 comparisons and DANN higher in 3; 1 is a tie. Comprehensive assessment… ranks DANN above M-CAP in all 5 comparisons that are not ties (5 are ties).

Comparisons in which each paper scores M-CAP or DANN higher3 other papers: M-CAP higher in 13, DANN higher in 3, 1 tie. Comprehensive assessment… (disagrees): M-CAP higher in 0, DANN higher in 5, 5 ties.3 other papers13 M-CAP higher131 tie3 DANN higher3Disagrees: Comprehensive assessment…Comprehensive assessment of computational algorithms in predicting cancer driver mutations5 tie55 DANN higher5
M-CAP higherTieDANN higher
Compared
DANN, M-CAP

Not reviewed yet.

SiPhy and phyloP: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks SiPhy above phyloP in 3 of 4 comparisons; 2 of 2 other studies rank phyloP above SiPhy.

2 papers rank phyloP above SiPhy in all 13 of their comparisons. Brandes et al. 2023 ranks SiPhy above phyloP in 3 of its 4 comparisons and phyloP higher in 1.

Comparisons in which each paper scores phyloP or SiPhy higher2 other papers: phyloP higher in 13, SiPhy higher in 0, 0 ties. Brandes et al. 2023 (disagrees): phyloP higher in 1, SiPhy higher in 3, 0 ties.2 other papers13 phyloP higher13Disagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods1 phyloP higher13 SiPhy higher3
phyloP higherTieSiPhy higher
Compared
SiPhy, phyloP

Not reviewed yet.

Geneformer and scGPT: Kedzierska et al. 2025, Additional file 1 (Supplementary Tables S1-S7, xlsx) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Kedzierska et al. 2025, Additional file 1 (Supplementary Tables S1-S7, xlsx) ranks scGPT above Geneformer in 83 of 89 comparisons; 2 of 2 other studies rank Geneformer above scGPT.

4 papers rank Geneformer above scGPT in 10 of their 19 comparisons and scGPT higher in 5; 4 are ties. Of those papers, Parameter-Efficient Fine-Tuning… ranks scGPT higher on its own (1 of 1). Kedzierska et al. 2025 ranks scGPT above Geneformer in 83 of its 97 comparisons and Geneformer higher in 6; 8 are ties.

Comparisons in which each paper scores Geneformer or scGPT higher4 other papers: Geneformer higher in 10, scGPT higher in 5, 4 ties. Kedzierska et al. 2025 (disagrees): Geneformer higher in 6, scGPT higher in 83, 8 ties.4 other papers10 Geneformer higher104 tie45 scGPT higher5Disagrees: Kedzierska et al. 2025Kedzierska et al. 2025, Additional file 1 (Supplementary Tables S1-S7, xlsx)6 Geneformer higher8 tie83 scGPT higher83
Geneformer higherTiescGPT higher
Compared
Geneformer, scGPT

Not reviewed yet.

Nucleotide Transformer and HyenaDNA: TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) ranks HyenaDNA above Nucleotide Transformer in 2 of 2 comparisons; 4 of 4 other studies rank Nucleotide Transformer above HyenaDNA.

5 papers rank Nucleotide Transformer above HyenaDNA in 77 of their 82 comparisons and HyenaDNA higher in 5. TraitGym official metrics ranks HyenaDNA above Nucleotide Transformer in all 2 of its comparisons.

Comparisons in which each paper scores Nucleotide Transformer or HyenaDNA higher5 other papers: Nucleotide Transformer higher in 77, HyenaDNA higher in 5, 0 ties. TraitGym official metrics (disagrees): Nucleotide Transformer higher in 0, HyenaDNA higher in 2, 0 ties.5 other papers77 Nucleotide Transformer higher775 HyenaDNA higherDisagrees: TraitGym official metricsTraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)2 HyenaDNA higher2
Nucleotide Transformer higherTieHyenaDNA higher
Compared
Nucleotide Transformer, HyenaDNA

Not reviewed yet.

Nucleotide Transformer and Evo: RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks Evo above Nucleotide Transformer in 3 of 4 comparisons; 2 of 2 other studies rank Nucleotide Transformer above Evo.

3 papers rank Nucleotide Transformer above Evo in 72 of their 145 comparisons and Evo higher in 65; 8 are ties. Of those papers, Genome modelling and design across all… ranks Evo higher on its own (61 of 90). RNAGym fitness leaderboard ranks Evo above Nucleotide Transformer in 3 of its 4 comparisons and Nucleotide Transformer higher in 1.

Comparisons in which each paper scores Nucleotide Transformer or Evo higher3 other papers: Nucleotide Transformer higher in 72, Evo higher in 65, 8 ties. RNAGym fitness leaderboard (disagrees): Nucleotide Transformer higher in 1, Evo higher in 3, 0 ties.3 other papers72 Nucleotide Transformer higher728 tie65 Evo higher65Disagrees: RNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md1 Nucleotide Transformer higher13 Evo higher3
Nucleotide Transformer higherTieEvo higher
Compared
Nucleotide Transformer, Evo

Not reviewed yet.

DNABERT-2 and Evo 2: BacBench LICENSE file at commit 64b7d45 reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

BacBench LICENSE file at commit 64b7d45 ranks DNABERT-2 above Evo 2 in 10 of 10 comparisons; 3 of 3 other studies rank Evo 2 above DNABERT-2.

4 papers rank Evo 2 above DNABERT-2 in 63 of their 80 comparisons and DNABERT-2 higher in 14; 3 are ties. BacBench LICENSE file at commit 64b7d45 ranks DNABERT-2 above Evo 2 in all 10 of its comparisons.

Comparisons in which each paper scores Evo 2 or DNABERT-2 higher4 other papers: Evo 2 higher in 63, DNABERT-2 higher in 14, 3 ties. BacBench LICENSE file at commit 64b7d45 (disagrees): Evo 2 higher in 0, DNABERT-2 higher in 10, 0 ties.4 other papers63 Evo 2 higher633 tie14 DNABERT-2 higher14Disagrees: BacBench LICENSE file at commit 64b7d45BacBench LICENSE file at commit 64b7d4510 DNABERT-2 higher10
Evo 2 higherTieDNABERT-2 higher
Compared
DNABERT-2, Evo 2

Not reviewed yet.

DNABERT-2 and Evo: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above DNABERT-2 in 6 of 6 comparisons; 2 of 2 other studies rank DNABERT-2 above Evo.

2 papers rank DNABERT-2 above Evo in 45 of their 55 comparisons and Evo higher in 10. Zero-shot benchmarking of RNA language… ranks Evo above DNABERT-2 in all 6 of its comparisons.

Comparisons in which each paper scores DNABERT-2 or Evo higher2 other papers: DNABERT-2 higher in 45, Evo higher in 10, 0 ties. Zero-shot benchmarking of RNA language… (disagrees): DNABERT-2 higher in 0, Evo higher in 6, 0 ties.2 other papers45 DNABERT-2 higher4510 Evo higher10Disagrees: Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning6 Evo higher6
DNABERT-2 higherTieEvo higher
Compared
DNABERT-2, Evo

Not reviewed yet.

FATHMM-XF and SIFT: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks SIFT above FATHMM-XF in 6 of 7 comparisons; 2 of 2 other studies rank FATHMM-XF above SIFT.

2 papers rank FATHMM-XF above SIFT in 25 of their 26 comparisons and SIFT higher in 1. Livesey and Marsh 2025 ranks SIFT above FATHMM-XF in 6 of its 7 comparisons and FATHMM-XF higher in 1.

Comparisons in which each paper scores FATHMM-XF or SIFT higher2 other papers: FATHMM-XF higher in 25, SIFT higher in 1, 0 ties. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 1, SIFT higher in 6, 0 ties.2 other papers25 FATHMM-XF higher251 SIFT higherDisagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 FATHMM-XF higher16 SIFT higher6
FATHMM-XF higherTieSIFT higher
Compared
FATHMM-XF, SIFT

Not reviewed yet.

CADD and Eigen: An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks CADD above Eigen in 6 of 6 comparisons; 2 of 2 other studies rank Eigen above CADD.

3 papers rank Eigen above CADD in 19 of their 22 comparisons and CADD higher in 3. An openly licensed benchmark… ranks CADD above Eigen in all 6 of its comparisons.

Comparisons in which each paper scores Eigen or CADD higher3 other papers: Eigen higher in 19, CADD higher in 3, 0 ties. An openly licensed benchmark… (disagrees): Eigen higher in 0, CADD higher in 6, 0 ties.3 other papers19 Eigen higher193 CADD higher3Disagrees: An openly licensed benchmark…An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers6 CADD higher6
Eigen higherTieCADD higher
Compared
CADD, Eigen

Not reviewed yet.

FATHMM-XF and PolyPhen-2: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks PolyPhen-2 above FATHMM-XF in 6 of 6 comparisons; 2 of 2 other studies rank FATHMM-XF above PolyPhen-2.

2 papers rank FATHMM-XF above PolyPhen-2 in 20 of their 26 comparisons and PolyPhen-2 higher in 1; 5 are ties. Livesey and Marsh 2025 ranks PolyPhen-2 above FATHMM-XF in all 6 of its comparisons.

Comparisons in which each paper scores FATHMM-XF or PolyPhen-2 higher2 other papers: FATHMM-XF higher in 20, PolyPhen-2 higher in 1, 5 ties. Livesey and Marsh 2025 (disagrees): FATHMM-XF higher in 0, PolyPhen-2 higher in 6, 0 ties.2 other papers20 FATHMM-XF higher205 tie51 PolyPhen-2 higherDisagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)6 PolyPhen-2 higher6
FATHMM-XF higherTiePolyPhen-2 higher
Compared
FATHMM-XF, PolyPhen-2

Not reviewed yet.

SiPhy and GERP: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • Same task, different protocols

Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks SiPhy above GERP in 4 of 4 comparisons; 2 of 2 other studies rank GERP above SiPhy.

2 papers rank GERP above SiPhy in 11 of their 13 comparisons and SiPhy higher in 1; 1 is a tie. Brandes et al. 2023 ranks SiPhy above GERP in all 4 of its comparisons.

Comparisons in which each paper scores GERP or SiPhy higher2 other papers: GERP higher in 11, SiPhy higher in 1, 1 tie. Brandes et al. 2023 (disagrees): GERP higher in 0, SiPhy higher in 4, 0 ties.2 other papers11 GERP higher111 tie1 SiPhy higherDisagrees: Brandes et al. 2023Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods4 SiPhy higher4
GERP higherTieSiPhy higher
Compared
SiPhy, GERP

Not reviewed yet.

MetaLR and MVP: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above MVP in 6 of 7 comparisons; 2 of 2 other studies rank MVP above MetaLR.

3 papers rank MVP above MetaLR in 18 of their 21 comparisons and MetaLR higher in 3. Livesey and Marsh 2025 ranks MetaLR above MVP in 6 of its 7 comparisons and MVP higher in 1.

Comparisons in which each paper scores MVP or MetaLR higher3 other papers: MVP higher in 18, MetaLR higher in 3, 0 ties. Livesey and Marsh 2025 (disagrees): MVP higher in 1, MetaLR higher in 6, 0 ties.3 other papers18 MVP higher183 MetaLR higher3Disagrees: Livesey and Marsh 2025Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML)1 MVP higher16 MetaLR higher6
MVP higherTieMetaLR higher
Compared
MetaLR, MVP

Not reviewed yet.

PrimateAI and PROVEAN: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks PrimateAI above PROVEAN in 5 of 6 comparisons; 2 of 2 other studies rank PROVEAN above PrimateAI.

4 papers rank PROVEAN above PrimateAI in 20 of their 34 comparisons and PrimateAI higher in 12; 2 are ties. Clinical likelihood ratios and balanced… ranks PrimateAI above PROVEAN in 5 of its 6 comparisons and PROVEAN higher in 1.

Comparisons in which each paper scores PROVEAN or PrimateAI higher4 other papers: PROVEAN higher in 20, PrimateAI higher in 12, 2 ties. Clinical likelihood ratios and balanced… (disagrees): PROVEAN higher in 1, PrimateAI higher in 5, 0 ties.4 other papers20 PROVEAN higher202 tie12 PrimateAI higher12Disagrees: Clinical likelihood ratios and balanced…Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes1 PROVEAN higher15 PrimateAI higher5
PROVEAN higherTiePrimateAI higher
Compared
PrimateAI, PROVEAN

Not reviewed yet.

MetaLR and MetaSVM: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MetaSVM above MetaLR in 4 of 5 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above MetaSVM in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

4 papers rank MetaLR above MetaSVM in 15 of their 39 comparisons and MetaSVM higher in 19; 5 are ties. Of those papers, Comprehensive assessment…, Clinical likelihood ratios and balanced… rank MetaSVM higher on their own (12 of 20, 4 of 6).

Comparisons in which each paper scores MetaLR or MetaSVM higher4 other papers: MetaLR higher in 15, MetaSVM higher in 19, 5 ties.4 other papers15 MetaLR higher155 tie519 MetaSVM higher19
MetaLR higherTieMetaSVM higher
Compared
MetaLR, MetaSVM

Not reviewed yet.

ESM-1v and ESM-2: Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) ranks ESM-2 above ESM-1v in 3 of 3 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1v above ESM-2 in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

3 papers rank ESM-1v above ESM-2 in 14 of their 31 comparisons and ESM-2 higher in 11; 6 are ties. Of those papers, Genomic heterogeneity inflates… ranks ESM-2 higher on its own (3 of 5).

Comparisons in which each paper scores ESM-1v or ESM-2 higher3 other papers: ESM-1v higher in 14, ESM-2 higher in 11, 6 ties.3 other papers14 ESM-1v higher146 tie611 ESM-2 higher11
ESM-1v higherTieESM-2 higher
Compared
ESM-1v, ESM-2

Not reviewed yet.

DEOGEN2 and MutFormer: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above DEOGEN2 in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks DEOGEN2 above MutFormer in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank DEOGEN2 above MutFormer in 6 of their 13 comparisons and MutFormer higher in 7. Of those papers, Livesey and Marsh 2025 ranks MutFormer higher on its own (7 of 7).

Comparisons in which each paper scores DEOGEN2 or MutFormer higher2 other papers: DEOGEN2 higher in 6, MutFormer higher in 7, 0 ties.2 other papers6 DEOGEN2 higher67 MutFormer higher7
DEOGEN2 higherTieMutFormer higher
Compared
DEOGEN2, MutFormer

Not reviewed yet.

CARP and VESPAl: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks VESPAl above CARP in 15 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above VESPAl in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank CARP above VESPAl in 10 of their 25 comparisons and VESPAl higher in 15. Of those papers, ProteinGym ranks VESPAl higher on its own (15 of 19).

Comparisons in which each paper scores CARP or VESPAl higher2 other papers: CARP higher in 10, VESPAl higher in 15, 0 ties.2 other papers10 CARP higher1015 VESPAl higher15
CARP higherTieVESPAl higher
Compared
CARP, VESPAl

Not reviewed yet.

SuSPect and MetaLR: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MetaLR above SuSPect in 4 of 5 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks SuSPect above MetaLR in 6 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank SuSPect above MetaLR in 7 of their 13 comparisons and MetaLR higher in 5; 1 is a tie. Of those papers, Clinical likelihood ratios and balanced… ranks MetaLR higher on its own (4 of 6).

Comparisons in which each paper scores SuSPect or MetaLR higher2 other papers: SuSPect higher in 7, MetaLR higher in 5, 1 tie.2 other papers7 SuSPect higher71 tie5 MetaLR higher5
SuSPect higherTieMetaLR higher
Compared
SuSPect, MetaLR

Not reviewed yet.

MetaLR and PrimateAI: Comprehensive assessment of computational algorithms in predicting cancer driver mutations and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks PrimateAI above MetaLR in 19 of 20 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above PrimateAI in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

4 papers rank MetaLR above PrimateAI in 14 of their 39 comparisons and PrimateAI higher in 25. Of those papers, Comprehensive assessment…, An openly licensed benchmark… rank PrimateAI higher on their own (19 of 20, 4 of 6).

Comparisons in which each paper scores MetaLR or PrimateAI higher4 other papers: MetaLR higher in 14, PrimateAI higher in 25, 0 ties.4 other papers14 MetaLR higher1425 PrimateAI higher25
MetaLR higherTiePrimateAI higher
Compared
MetaLR, PrimateAI

Not reviewed yet.

MetaRNN and VARITY: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks VARITY above MetaRNN in 6 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks MetaRNN above VARITY in 3 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

3 papers rank MetaRNN above VARITY in 5 of their 14 comparisons and VARITY higher in 7; 2 are ties. Of those papers, Livesey and Marsh 2025 ranks VARITY higher on its own (6 of 7).

Comparisons in which each paper scores MetaRNN or VARITY higher3 other papers: MetaRNN higher in 5, VARITY higher in 7, 2 ties.3 other papers5 MetaRNN higher52 tie27 VARITY higher7
MetaRNN higherTieVARITY higher
Compared
MetaRNN, VARITY

Not reviewed yet.

MPC and MutFormer: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above MPC in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks MPC above MutFormer in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank MPC above MutFormer in 6 of their 13 comparisons and MutFormer higher in 7. Of those papers, Livesey and Marsh 2025 ranks MutFormer higher on its own (7 of 7).

Comparisons in which each paper scores MPC or MutFormer higher2 other papers: MPC higher in 6, MutFormer higher in 7, 0 ties.2 other papers6 MPC higher67 MutFormer higher7
MPC higherTieMutFormer higher
Compared
MPC, MutFormer

Not reviewed yet.

ESM-1b and Tranception: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above ESM-1b in 15 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above Tranception in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank ESM-1b above Tranception in 11 of their 26 comparisons and Tranception higher in 15. Of those papers, ProteinGym ranks Tranception higher on its own (15 of 19).

Comparisons in which each paper scores ESM-1b or Tranception higher2 other papers: ESM-1b higher in 11, Tranception higher in 15, 0 ties.2 other papers11 ESM-1b higher1115 Tranception higher15
ESM-1b higherTieTranception higher
Compared
ESM-1b, Tranception

Not reviewed yet.

Boltz-2 and Protenix: Shen et al. 2025, Chemical Science (Europe PMC full text) and POISK Explorer project site disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

Shen et al. 2025, Chemical Science (Europe PMC full text) ranks Boltz-2 above Protenix in 26 of 28 comparisons; POISK Explorer project site ranks Protenix above Boltz-2 in 10 of 10 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank Boltz-2 above Protenix in 26 of their 42 comparisons and Protenix higher in 12; 4 are ties. Of those papers, POISK Explorer project site ranks Protenix higher on its own (10 of 10).

Comparisons in which each paper scores Boltz-2 or Protenix higher2 other papers: Boltz-2 higher in 26, Protenix higher in 12, 4 ties.2 other papers26 Boltz-2 higher264 tie412 Protenix higher12
Boltz-2 higherTieProtenix higher
Compared
Boltz-2, Protenix

Not reviewed yet.

SNAP2 and FATHMM-MKL: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks FATHMM-MKL above SNAP2 in 3 of 4 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks SNAP2 above FATHMM-MKL in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank SNAP2 above FATHMM-MKL in 8 of their 12 comparisons and FATHMM-MKL higher in 3; 1 is a tie. Of those papers, Clinical likelihood ratios and balanced… ranks FATHMM-MKL higher on its own (3 of 5).

Comparisons in which each paper scores SNAP2 or FATHMM-MKL higher2 other papers: SNAP2 higher in 8, FATHMM-MKL higher in 3, 1 tie.2 other papers8 SNAP2 higher81 tie3 FATHMM-MKL higher3
SNAP2 higherTieFATHMM-MKL higher
Compared
SNAP2, FATHMM-MKL

Not reviewed yet.

Nucleotide Transformer and RNAErnie: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks Nucleotide Transformer above RNAErnie in 7 of 7 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above Nucleotide Transformer in 4 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank Nucleotide Transformer above RNAErnie in 7 of their 19 comparisons and RNAErnie higher in 4; 8 are ties. Of those papers, RNAGym fitness leaderboard ranks RNAErnie higher on its own (4 of 4).

Comparisons in which each paper scores Nucleotide Transformer or RNAErnie higher2 other papers: Nucleotide Transformer higher in 7, RNAErnie higher in 4, 8 ties.2 other papers7 Nucleotide Transformer higher78 tie84 RNAErnie higher4
Nucleotide Transformer higherTieRNAErnie higher
Compared
Nucleotide Transformer, RNAErnie

Not reviewed yet.

DNABERT-2 and PhyloGPN: GUANinE v1.1 reveals complementarity of supervised and genomic language models and Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks DNABERT-2 above PhyloGPN in 6 of 6 comparisons; Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) ranks PhyloGPN above DNABERT-2 in 14 of 14 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank DNABERT-2 above PhyloGPN in 6 of their 20 comparisons and PhyloGPN higher in 14. Of those papers, Genomic heterogeneity inflates… ranks PhyloGPN higher on its own (14 of 14).

Comparisons in which each paper scores DNABERT-2 or PhyloGPN higher2 other papers: DNABERT-2 higher in 6, PhyloGPN higher in 14, 0 ties.2 other papers6 DNABERT-2 higher614 PhyloGPN higher14
DNABERT-2 higherTiePhyloGPN higher
Compared
DNABERT-2, PhyloGPN

Not reviewed yet.

MutFormer and SIFT: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above SIFT in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks SIFT above MutFormer in 3 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank MutFormer above SIFT in 8 of their 13 comparisons and SIFT higher in 3; 2 are ties. Of those papers, An openly licensed benchmark… ranks SIFT higher on its own (3 of 6).

Comparisons in which each paper scores MutFormer or SIFT higher2 other papers: MutFormer higher in 8, SIFT higher in 3, 2 ties.2 other papers8 MutFormer higher82 tie23 SIFT higher3
MutFormer higherTieSIFT higher
Compared
MutFormer, SIFT

Not reviewed yet.

DNABERT and RNA-MSM: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks DNABERT above RNA-MSM in 4 of 4 comparisons; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks RNA-MSM above DNABERT in 8 of 8 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank DNABERT above RNA-MSM in 4 of their 20 comparisons and RNA-MSM higher in 8; 8 are ties. Of those papers, Zero-shot benchmarking of RNA language… ranks RNA-MSM higher on its own (8 of 8).

Comparisons in which each paper scores DNABERT or RNA-MSM higher2 other papers: DNABERT higher in 4, RNA-MSM higher in 8, 8 ties.2 other papers4 DNABERT higher48 tie88 RNA-MSM higher8
DNABERT higherTieRNA-MSM higher
Compared
DNABERT, RNA-MSM

Not reviewed yet.

ChromBPNet and Caduceus: GUANinE v1.1 reveals complementarity of supervised and genomic language models and Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks Caduceus above ChromBPNet in 6 of 6 comparisons; Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants ranks ChromBPNet above Caduceus in 4 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank ChromBPNet above Caduceus in 4 of their 10 comparisons and Caduceus higher in 6. Of those papers, GUANinE v1.1 reveals complementarity… ranks Caduceus higher on its own (6 of 6).

Comparisons in which each paper scores ChromBPNet or Caduceus higher2 other papers: ChromBPNet higher in 4, Caduceus higher in 6, 0 ties.2 other papers4 ChromBPNet higher46 Caduceus higher6
ChromBPNet higherTieCaduceus higher
Compared
ChromBPNet, Caduceus

Not reviewed yet.

CARP and VESPA: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks VESPA above CARP in 18 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above VESPA in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank CARP above VESPA in 7 of their 25 comparisons and VESPA higher in 18. Of those papers, ProteinGym ranks VESPA higher on its own (18 of 19).

Comparisons in which each paper scores CARP or VESPA higher2 other papers: CARP higher in 7, VESPA higher in 18, 0 ties.2 other papers7 CARP higher718 VESPA higher18
CARP higherTieVESPA higher
Compared
CARP, VESPA

Not reviewed yet.

Condel and MetaSVM: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MetaSVM above Condel in 4 of 5 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks Condel above MetaSVM in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank Condel above MetaSVM in 8 of their 13 comparisons and MetaSVM higher in 4; 1 is a tie. Of those papers, Clinical likelihood ratios and balanced… ranks MetaSVM higher on its own (4 of 6).

Comparisons in which each paper scores Condel or MetaSVM higher2 other papers: Condel higher in 8, MetaSVM higher in 4, 1 tie.2 other papers8 Condel higher81 tie4 MetaSVM higher4
Condel higherTieMetaSVM higher
Compared
Condel, MetaSVM

Not reviewed yet.

ESM-2 and Tranception: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above ESM-2 in 16 of 18 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-2 above Tranception in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank ESM-2 above Tranception in 9 of their 26 comparisons and Tranception higher in 16; 1 is a tie. Of those papers, ProteinGym ranks Tranception higher on its own (16 of 19).

Comparisons in which each paper scores ESM-2 or Tranception higher2 other papers: ESM-2 higher in 9, Tranception higher in 16, 1 tie.2 other papers9 ESM-2 higher91 tie16 Tranception higher16
ESM-2 higherTieTranception higher
Compared
ESM-2, Tranception

Not reviewed yet.

ESM-1v and DeepSequence: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks DeepSequence above ESM-1v in 13 of 17 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1v above DeepSequence in 6 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank ESM-1v above DeepSequence in 10 of their 26 comparisons and DeepSequence higher in 14; 2 are ties. Of those papers, ProteinGym ranks DeepSequence higher on its own (13 of 19).

Comparisons in which each paper scores ESM-1v or DeepSequence higher2 other papers: ESM-1v higher in 10, DeepSequence higher in 14, 2 ties.2 other papers10 ESM-1v higher102 tie14 DeepSequence higher14
ESM-1v higherTieDeepSequence higher
Compared
ESM-1v, DeepSequence

Not reviewed yet.

Condel and SIFT: Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks SIFT above Condel in 5 of 5 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks Condel above SIFT in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank Condel above SIFT in 7 of their 12 comparisons and SIFT higher in 5. Of those papers, Clinical likelihood ratios and balanced… ranks SIFT higher on its own (5 of 5).

Comparisons in which each paper scores Condel or SIFT higher2 other papers: Condel higher in 7, SIFT higher in 5, 0 ties.2 other papers7 Condel higher75 SIFT higher5
Condel higherTieSIFT higher
Compared
Condel, SIFT

Not reviewed yet.

LRT and PolyPhen-2: Comprehensive assessment of computational algorithms in predicting cancer driver mutations and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks LRT above PolyPhen-2 in 14 of 18 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks PolyPhen-2 above LRT in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

3 papers rank LRT above PolyPhen-2 in 17 of their 32 comparisons and PolyPhen-2 higher in 13; 2 are ties. Of those papers, Livesey and Marsh 2025 ranks PolyPhen-2 higher on its own (6 of 6).

Comparisons in which each paper scores LRT or PolyPhen-2 higher3 other papers: LRT higher in 17, PolyPhen-2 higher in 13, 2 ties.3 other papers17 LRT higher172 tie13 PolyPhen-2 higher13
LRT higherTiePolyPhen-2 higher
Compared
LRT, PolyPhen-2

Not reviewed yet.

DNABERT and UTR-LM: Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications and Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks DNABERT above UTR-LM in 8 of 8 comparisons; Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks UTR-LM above DNABERT in 25 of 26 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank DNABERT above UTR-LM in 9 of their 42 comparisons and UTR-LM higher in 25; 8 are ties. Of those papers, Zero-shot benchmarking of RNA language… ranks UTR-LM higher on its own (25 of 30).

Comparisons in which each paper scores DNABERT or UTR-LM higher2 other papers: DNABERT higher in 9, UTR-LM higher in 25, 8 ties.2 other papers9 DNABERT higher98 tie825 UTR-LM higher25
DNABERT higherTieUTR-LM higher
Compared
DNABERT, UTR-LM

Not reviewed yet.

CARP and ESM-1b: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM-1b above CARP in 14 of 18 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above ESM-1b in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank CARP above ESM-1b in 10 of their 25 comparisons and ESM-1b higher in 14; 1 is a tie. Of those papers, ProteinGym ranks ESM-1b higher on its own (14 of 19).

Comparisons in which each paper scores CARP or ESM-1b higher2 other papers: CARP higher in 10, ESM-1b higher in 14, 1 tie.2 other papers10 CARP higher101 tie14 ESM-1b higher14
CARP higherTieESM-1b higher
Compared
CARP, ESM-1b

Not reviewed yet.

CARP and Tranception: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above CARP in 19 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above Tranception in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank CARP above Tranception in 6 of their 25 comparisons and Tranception higher in 19. Of those papers, ProteinGym ranks Tranception higher on its own (19 of 19).

Comparisons in which each paper scores CARP or Tranception higher2 other papers: CARP higher in 6, Tranception higher in 19, 0 ties.2 other papers6 CARP higher619 Tranception higher19
CARP higherTieTranception higher
Compared
CARP, Tranception

Not reviewed yet.

MetaRNN and MutScore: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaRNN above MutScore in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks MutScore above MetaRNN in 3 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank MetaRNN above MutScore in 8 of their 13 comparisons and MutScore higher in 3; 2 are ties. Of those papers, An openly licensed benchmark… ranks MutScore higher on its own (3 of 6).

Comparisons in which each paper scores MetaRNN or MutScore higher2 other papers: MetaRNN higher in 8, MutScore higher in 3, 2 ties.2 other papers8 MetaRNN higher82 tie23 MutScore higher3
MetaRNN higherTieMutScore higher
Compared
MetaRNN, MutScore

Not reviewed yet.

MutFormer and PolyPhen-2: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above PolyPhen-2 in 6 of 6 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks PolyPhen-2 above MutFormer in 5 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

2 papers rank MutFormer above PolyPhen-2 in 7 of their 12 comparisons and PolyPhen-2 higher in 5. Of those papers, An openly licensed benchmark… ranks PolyPhen-2 higher on its own (5 of 6).

Comparisons in which each paper scores MutFormer or PolyPhen-2 higher2 other papers: MutFormer higher in 7, PolyPhen-2 higher in 5, 0 ties.2 other papers7 MutFormer higher75 PolyPhen-2 higher5
MutFormer higherTiePolyPhen-2 higher
Compared
MutFormer, PolyPhen-2

Not reviewed yet.

ClinPred and MetaRNN: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaRNN above ClinPred in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks ClinPred above MetaRNN in 2 of 2 comparisons. No other study gives a verdict, so neither is the outlier.

3 papers rank ClinPred above MetaRNN in 2 of their 14 comparisons and MetaRNN higher in 7; 5 are ties. Of those papers, Livesey and Marsh 2025 ranks MetaRNN higher on its own (7 of 7).

Comparisons in which each paper scores ClinPred or MetaRNN higher3 other papers: ClinPred higher in 2, MetaRNN higher in 7, 5 ties.3 other papers2 ClinPred higher25 tie57 MetaRNN higher7
ClinPred higherTieMetaRNN higher
Compared
ClinPred, MetaRNN

Not reviewed yet.

First page · Next page