Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.
These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.
A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.
mRNABench: A curated benchmark for mature mRNA property and function prediction ranks DNABERT-2 above RNA-FM in 7 of 7 comparisons; other studies rank RNA-FM above DNABERT-2 in 30 of 30.
2 papers rank RNA-FM above DNABERT-2 in all 30 comparisons that are not ties (4 are ties). MRNABench ranks DNABERT-2 above RNA-FM in all 7 comparisons that are not ties (3 are ties).
Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) ranks Geneformer above scGPT in 3 of 4 comparisons; other studies rank scGPT above Geneformer in 95 of 98.
3 papers rank scGPT above Geneformer in 95 of their 100 comparisons and Geneformer higher in 3; 2 are ties. Of those papers, Deep-learning-based gene perturbation… ranks Geneformer higher on its own (2 of 2). Wu et al. 2025 ranks Geneformer above scGPT in 3 of its 4 comparisons and scGPT higher in 1.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above Eigen in 6 of 7 comparisons; other studies rank Eigen above MutationAssessor in 21 of 21.
3 papers rank Eigen above MutationAssessor in all 21 comparisons that are not ties (1 is a tie). Livesey and Marsh 2025 ranks MutationAssessor above Eigen in 6 of its 7 comparisons and Eigen higher in 1.
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks ESM-1b above EVE in 6 of 6 comparisons; other studies rank EVE above ESM-1b in 23 of 24.
2 papers rank EVE above ESM-1b in 23 of their 27 comparisons and ESM-1b higher in 1; 3 are ties. Brandes et al. 2023 ranks ESM-1b above EVE in all 6 comparisons that are not ties (2 are ties).
pfmbench primary benchmark evidence ranks ESM-2 above ESM3 in 9 of 12 comparisons; other studies rank ESM3 above ESM-2 in 17 of 18.
ProteinGym AMFR assay-level Spearman… ranks ESM3 above ESM-2 in 17 of its 19 comparisons and ESM-2 higher in 1; 1 is a tie. Pfmbench primary benchmark evidence ranks ESM-2 above ESM3 in 9 of its 12 comparisons and ESM3 higher in 3.
ProteinGym AMFR assay-level Spearman comparison ranks ESM-2 above ESM-1v in 15 of 17 comparisons; other studies rank ESM-1v above ESM-2 in 10 of 11.
2 papers rank ESM-1v above ESM-2 in 10 of their 12 comparisons and ESM-2 higher in 1; 1 is a tie. ProteinGym AMFR assay-level Spearman… ranks ESM-2 above ESM-1v in 15 of its 19 comparisons and ESM-1v higher in 2; 2 are ties.
Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines ranks Geneformer above scGPT in 2 of 2 comparisons; other studies rank scGPT above Geneformer in 96 of 100.
3 papers rank scGPT above Geneformer in 96 of their 102 comparisons and Geneformer higher in 4; 2 are ties. Of those papers, Wu et al. 2025 ranks Geneformer higher on its own (3 of 4). Deep-learning-based gene perturbation… ranks Geneformer above scGPT in all 2 of its comparisons.
ClairS-TO: a deep-learning method for long-read tumor-only somatic small variant calling ranks Mutect2 above Octopus in 14 of 14 comparisons; other studies rank Octopus above Mutect2 in 154 of 154.
Achieving robust somatic mutation… ranks Octopus above Mutect2 in all 154 of its comparisons. ClairS-TO ranks Mutect2 above Octopus in all 14 of its comparisons.
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above AIDO.RNA in 3 of 4 comparisons; other studies rank AIDO.RNA above RNAErnie in 36 of 36.
Zero-shot benchmarking of RNA language… ranks AIDO.RNA above RNAErnie in all 36 of its comparisons. RNAGym fitness leaderboard ranks RNAErnie above AIDO.RNA in 3 of its 4 comparisons and AIDO.RNA higher in 1.
A benchmarking study of individual somatic variant callers and voting-based ensembles for whole-exome sequencing ranks DeepSomatic above Majority-vote consensus of somatic callers in 8 of 10 comparisons; other studies rank Majority-vote consensus of somatic callers above DeepSomatic in 20 of 20.
Prospectively frozen external… ranks Majority-vote consensus of somatic callers above DeepSomatic in all 20 comparisons that are not ties (4 are ties). A benchmarking study of individual… ranks DeepSomatic above Majority-vote consensus of somatic callers in 8 of its 12 comparisons and Majority-vote consensus of somatic callers higher in 2; 2 are ties.
Majority-vote consensus of somatic callers higherTieDeepSomatic higher
proteinbench primary benchmark evidence ranks ProteinMPNN above ESM3 in 10 of 10 comparisons; other studies rank ESM3 above ProteinMPNN in 18 of 19.
ProteinGym AMFR assay-level Spearman… ranks ESM3 above ProteinMPNN in 18 of its 19 comparisons and ProteinMPNN higher in 1. Proteinbench primary benchmark evidence ranks ProteinMPNN above ESM3 in all 10 of its comparisons.
ProteinGym AMFR assay-level Spearman comparison ranks ESM3 above ProteinMPNN in 18 of 19 comparisons; other studies rank ProteinMPNN above ESM3 in 10 of 10.
Proteinbench primary benchmark evidence ranks ProteinMPNN above ESM3 in all 10 of its comparisons. ProteinGym AMFR assay-level Spearman… ranks ESM3 above ProteinMPNN in 18 of its 19 comparisons and ProteinMPNN higher in 1.