Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.
These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.
A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.
proteinbench primary benchmark evidence ranks ProteinMPNN above ESM-IF1 in 10 of 10 comparisons; other studies rank ESM-IF1 above ProteinMPNN in 24 of 24.
2 papers rank ESM-IF1 above ProteinMPNN in all 24 of their comparisons. Proteinbench primary benchmark evidence ranks ProteinMPNN above ESM-IF1 in all 10 of its comparisons.
mRNABench: A curated benchmark for mature mRNA property and function prediction ranks DNABERT-2 above RNA-FM in 7 of 7 comparisons; other studies rank RNA-FM above DNABERT-2 in 30 of 30.
2 papers rank RNA-FM above DNABERT-2 in all 30 comparisons that are not ties (4 are ties). MRNABench ranks DNABERT-2 above RNA-FM in all 7 comparisons that are not ties (3 are ties).
Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) ranks Geneformer above scGPT in 3 of 4 comparisons; other studies rank scGPT above Geneformer in 95 of 98.
3 papers rank scGPT above Geneformer in 95 of their 100 comparisons and Geneformer higher in 3; 2 are ties. Of those papers, Deep-learning-based gene perturbation… ranks Geneformer higher on its own (2 of 2). Wu et al. 2025 ranks Geneformer above scGPT in 3 of its 4 comparisons and scGPT higher in 1.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutationAssessor above Eigen in 6 of 7 comparisons; other studies rank Eigen above MutationAssessor in 21 of 21.
3 papers rank Eigen above MutationAssessor in all 21 comparisons that are not ties (1 is a tie). Livesey and Marsh 2025 ranks MutationAssessor above Eigen in 6 of its 7 comparisons and Eigen higher in 1.
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks ESM-1b above EVE in 6 of 6 comparisons; other studies rank EVE above ESM-1b in 23 of 24.
2 papers rank EVE above ESM-1b in 23 of their 27 comparisons and ESM-1b higher in 1; 3 are ties. Brandes et al. 2023 ranks ESM-1b above EVE in all 6 comparisons that are not ties (2 are ties).
pfmbench primary benchmark evidence ranks ESM-2 above ESM3 in 9 of 12 comparisons; other studies rank ESM3 above ESM-2 in 17 of 18.
ProteinGym AMFR assay-level Spearman… ranks ESM3 above ESM-2 in 17 of its 19 comparisons and ESM-2 higher in 1; 1 is a tie. Pfmbench primary benchmark evidence ranks ESM-2 above ESM3 in 9 of its 12 comparisons and ESM3 higher in 3.
ProteinGym AMFR assay-level Spearman comparison ranks ESM-2 above ESM-1v in 15 of 17 comparisons; other studies rank ESM-1v above ESM-2 in 10 of 11.
2 papers rank ESM-1v above ESM-2 in 10 of their 12 comparisons and ESM-2 higher in 1; 1 is a tie. ProteinGym AMFR assay-level Spearman… ranks ESM-2 above ESM-1v in 15 of its 19 comparisons and ESM-1v higher in 2; 2 are ties.
Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines ranks Geneformer above scGPT in 2 of 2 comparisons; other studies rank scGPT above Geneformer in 96 of 100.
3 papers rank scGPT above Geneformer in 96 of their 102 comparisons and Geneformer higher in 4; 2 are ties. Of those papers, Wu et al. 2025 ranks Geneformer higher on its own (3 of 4). Deep-learning-based gene perturbation… ranks Geneformer above scGPT in all 2 of its comparisons.
ClairS-TO: a deep-learning method for long-read tumor-only somatic small variant calling ranks Mutect2 above Octopus in 14 of 14 comparisons; other studies rank Octopus above Mutect2 in 154 of 154.
Achieving robust somatic mutation… ranks Octopus above Mutect2 in all 154 of its comparisons. ClairS-TO ranks Mutect2 above Octopus in all 14 of its comparisons.
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above AIDO.RNA in 3 of 4 comparisons; other studies rank AIDO.RNA above RNAErnie in 36 of 36.
Zero-shot benchmarking of RNA language… ranks AIDO.RNA above RNAErnie in all 36 of its comparisons. RNAGym fitness leaderboard ranks RNAErnie above AIDO.RNA in 3 of its 4 comparisons and AIDO.RNA higher in 1.
A benchmarking study of individual somatic variant callers and voting-based ensembles for whole-exome sequencing ranks DeepSomatic above Majority-vote consensus of somatic callers in 8 of 10 comparisons; other studies rank Majority-vote consensus of somatic callers above DeepSomatic in 20 of 20.
Prospectively frozen external… ranks Majority-vote consensus of somatic callers above DeepSomatic in all 20 comparisons that are not ties (4 are ties). A benchmarking study of individual… ranks DeepSomatic above Majority-vote consensus of somatic callers in 8 of its 12 comparisons and Majority-vote consensus of somatic callers higher in 2; 2 are ties.
Majority-vote consensus of somatic callers higherTieDeepSomatic higher
proteinbench primary benchmark evidence ranks ProteinMPNN above ESM3 in 10 of 10 comparisons; other studies rank ESM3 above ProteinMPNN in 18 of 19.
ProteinGym AMFR assay-level Spearman… ranks ESM3 above ProteinMPNN in 18 of its 19 comparisons and ProteinMPNN higher in 1. Proteinbench primary benchmark evidence ranks ProteinMPNN above ESM3 in all 10 of its comparisons.
ProteinGym AMFR assay-level Spearman comparison ranks ESM3 above ProteinMPNN in 18 of 19 comparisons; other studies rank ProteinMPNN above ESM3 in 10 of 10.
Proteinbench primary benchmark evidence ranks ProteinMPNN above ESM3 in all 10 of its comparisons. ProteinGym AMFR assay-level Spearman… ranks ESM3 above ProteinMPNN in 18 of its 19 comparisons and ProteinMPNN higher in 1.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks PHACTboost above BayesDel in 7 of 7 comparisons; other studies rank BayesDel above PHACTboost in 4 of 5.
An openly licensed benchmark… ranks BayesDel above PHACTboost in 4 of its 6 comparisons and PHACTboost higher in 1; 1 is a tie. Livesey and Marsh 2025 ranks PHACTboost above BayesDel in all 7 of its comparisons.
Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks LRT above MetaLR in 15 of 17 comparisons; other studies rank MetaLR above LRT in 12 of 13.
2 papers rank MetaLR above LRT in 12 of their 13 comparisons and LRT higher in 1. Comprehensive assessment… ranks LRT above MetaLR in 15 of its 20 comparisons and MetaLR higher in 2; 3 are ties.
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks GERP above GenoCanyon in 4 of 5 comparisons; other studies rank GenoCanyon above GERP in 9 of 11.
2 papers rank GenoCanyon above GERP in 9 of their 11 comparisons and GERP higher in 2. Clinical likelihood ratios and balanced… ranks GERP above GenoCanyon in 4 of its 6 comparisons and GenoCanyon higher in 1; 1 is a tie.
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks MetaSVM above MutFormer in 4 of 5 comparisons; other studies rank MutFormer above MetaSVM in 7 of 7.
Livesey and Marsh 2025 ranks MutFormer above MetaSVM in all 7 of its comparisons. An openly licensed benchmark… ranks MetaSVM above MutFormer in 4 of its 6 comparisons and MutFormer higher in 1; 1 is a tie.
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks GERP above FATHMM-MKL in 3 of 4 comparisons; other studies rank FATHMM-MKL above GERP in 6 of 6.
Livesey and Marsh 2025 ranks FATHMM-MKL above GERP in all 6 comparisons that are not ties (1 is a tie). Clinical likelihood ratios and balanced… ranks GERP above FATHMM-MKL in 3 of its 5 comparisons and FATHMM-MKL higher in 1; 1 is a tie.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above CADD in 6 of 7 comparisons; other studies rank CADD above MetaLR in 19 of 21.
3 papers rank CADD above MetaLR in 19 of their 21 comparisons and MetaLR higher in 2. Livesey and Marsh 2025 ranks MetaLR above CADD in 6 of its 7 comparisons and CADD higher in 1.
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks CADD above Condel in 4 of 5 comparisons; other studies rank Condel above CADD in 6 of 7.
Livesey and Marsh 2025 ranks Condel above CADD in 6 of its 7 comparisons and CADD higher in 1. Clinical likelihood ratios and balanced… ranks CADD above Condel in 4 of its 5 comparisons and Condel higher in 1.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1v above Tranception in 7 of 7 comparisons; other studies rank Tranception above ESM-1v in 14 of 17.
ProteinGym AMFR assay-level Spearman… ranks Tranception above ESM-1v in 14 of its 19 comparisons and ESM-1v higher in 3; 2 are ties. Livesey and Marsh 2025 ranks ESM-1v above Tranception in all 7 of its comparisons.
Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks FATHMM-XF above CADD in 10 of 10 comparisons; other studies rank CADD above FATHMM-XF in 12 of 12.
2 papers rank CADD above FATHMM-XF in all 12 comparisons that are not ties (1 is a tie). Comprehensive assessment… ranks FATHMM-XF above CADD in all 10 of its comparisons.
Benchmarking pre-trained genomic language models for RNA sequence-related predictive applications ranks DNABERT above RNA-MSM in 4 of 4 comparisons; other studies rank RNA-MSM above DNABERT in 8 of 8.
Zero-shot benchmarking of RNA language… ranks RNA-MSM above DNABERT in all 8 of its comparisons. Benchmarking pre-trained genomic… ranks DNABERT above RNA-MSM in all 4 comparisons that are not ties (8 are ties).
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks MVP above ESM-1b in 2 of 2 comparisons; other studies rank ESM-1b above MVP in 13 of 13.
2 papers rank ESM-1b above MVP in all 13 of their comparisons. Brandes et al. 2023 ranks MVP above ESM-1b in all 2 comparisons that are not ties (2 are ties).
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above RiNALMo in 3 of 4 comparisons; other studies rank RiNALMo above RNAErnie in 29 of 35.
Zero-shot benchmarking of RNA language… ranks RiNALMo above RNAErnie in 29 of its 36 comparisons and RNAErnie higher in 6; 1 is a tie. RNAGym fitness leaderboard ranks RNAErnie above RiNALMo in 3 of its 4 comparisons and RiNALMo higher in 1.
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants ranks ChromBPNet above Caduceus in 4 of 4 comparisons; other studies rank Caduceus above ChromBPNet in 6 of 6.
GUANinE v1.1 reveals complementarity… ranks Caduceus above ChromBPNet in all 6 of its comparisons. Comparative Analysis of Deep Learning… ranks ChromBPNet above Caduceus in all 4 of its comparisons.
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks FATHMM-XF above MutFormer in 6 of 6 comparisons; other studies rank MutFormer above FATHMM-XF in 7 of 7.
Livesey and Marsh 2025 ranks MutFormer above FATHMM-XF in all 7 of its comparisons. An openly licensed benchmark… ranks FATHMM-XF above MutFormer in all 6 of its comparisons.
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes ranks MVP above Condel in 4 of 5 comparisons; other studies rank Condel above MVP in 6 of 7.
Livesey and Marsh 2025 ranks Condel above MVP in 6 of its 7 comparisons and MVP higher in 1. Clinical likelihood ratios and balanced… ranks MVP above Condel in 4 of its 5 comparisons and Condel higher in 1.
Comprehensive assessment of computational algorithms in predicting cancer driver mutations ranks FATHMM-XF above PolyPhen-2 in 14 of 16 comparisons; other studies rank PolyPhen-2 above FATHMM-XF in 9 of 11.
2 papers rank PolyPhen-2 above FATHMM-XF in 9 of their 12 comparisons and FATHMM-XF higher in 2; 1 is a tie. Comprehensive assessment… ranks FATHMM-XF above PolyPhen-2 in 14 of its 20 comparisons and PolyPhen-2 higher in 2; 4 are ties.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1v above MSA Transformer in 5 of 5 comparisons; other studies rank MSA Transformer above ESM-1v in 16 of 19.
ProteinGym AMFR assay-level Spearman… ranks MSA Transformer above ESM-1v in 16 of its 19 comparisons and ESM-1v higher in 3. Livesey and Marsh 2025 ranks ESM-1v above MSA Transformer in all 5 comparisons that are not ties (1 is a tie).
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks RNA-FM above RNAErnie in 28 of 36 comparisons; other studies rank RNAErnie above RNA-FM in 6 of 6.
2 papers rank RNAErnie above RNA-FM in all 6 comparisons that are not ties (13 are ties). Zero-shot benchmarking of RNA language… ranks RNA-FM above RNAErnie in 28 of its 36 comparisons and RNAErnie higher in 8.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks Condel above GERP in 7 of 7 comparisons; other studies rank GERP above Condel in 6 of 6.
Clinical likelihood ratios and balanced… ranks GERP above Condel in all 6 of its comparisons. Livesey and Marsh 2025 ranks Condel above GERP in all 7 of its comparisons.
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks AIDO.RNA above Evo in 4 of 4 comparisons; other studies rank Evo above AIDO.RNA in 5 of 5.
Zero-shot benchmarking of RNA language… ranks Evo above AIDO.RNA in all 5 comparisons that are not ties (1 is a tie). RNAGym fitness leaderboard ranks AIDO.RNA above Evo in all 4 of its comparisons.
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above Evo in 4 of 4 comparisons; other studies rank Evo above RNAErnie in 6 of 6.
Zero-shot benchmarking of RNA language… ranks Evo above RNAErnie in all 6 of its comparisons. RNAGym fitness leaderboard ranks RNAErnie above Evo in all 4 of its comparisons.
ProteinGym AMFR assay-level Spearman comparison ranks Tranception above ESM-1b in 16 of 19 comparisons; other studies rank ESM-1b above Tranception in 7 of 7.
Livesey and Marsh 2025 ranks ESM-1b above Tranception in all 7 of its comparisons. ProteinGym AMFR assay-level Spearman… ranks Tranception above ESM-1b in 16 of its 19 comparisons and ESM-1b higher in 3.
Taxonomic classification of DNA sequences beyond sequence similarity using deep neural networks (PNAS PDF) ranks Kraken2 above sourmash in 7 of 9 comparisons; other studies rank sourmash above Kraken2 in 24 of 29.
Evaluation of taxonomic classification… ranks sourmash above Kraken2 in 24 of its 48 comparisons and Kraken2 higher in 5; 19 are ties. Taxonomic classification of DNA… ranks Kraken2 above sourmash in 7 of its 11 comparisons and sourmash higher in 2; 2 are ties.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks PHACTboost above MutScore in 7 of 7 comparisons; other studies rank MutScore above PHACTboost in 5 of 5.
An openly licensed benchmark… ranks MutScore above PHACTboost in all 5 comparisons that are not ties (1 is a tie). Livesey and Marsh 2025 ranks PHACTboost above MutScore in all 7 of its comparisons.
Transfer learning to leverage larger datasets for improved prediction of protein stability changes ranks MUpro above ThermoNet in 5 of 6 comparisons; other studies rank ThermoNet above MUpro in 9 of 9.
Predicting protein stability changes… ranks ThermoNet above MUpro in all 9 comparisons that are not ties (1 is a tie). Transfer learning to leverage larger… ranks MUpro above ThermoNet in 5 of its 6 comparisons and ThermoNet higher in 1.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above PrimateAI in 7 of 7 comparisons; other studies rank PrimateAI above MutFormer in 6 of 6.
An openly licensed benchmark… ranks PrimateAI above MutFormer in all 6 of its comparisons. Livesey and Marsh 2025 ranks MutFormer above PrimateAI in all 7 of its comparisons.
ProteinGym AMFR assay-level Spearman comparison ranks Tranception above EVmutation in 15 of 18 comparisons; other studies rank EVmutation above Tranception in 7 of 7.
Livesey and Marsh 2025 ranks EVmutation above Tranception in all 7 of its comparisons. ProteinGym AMFR assay-level Spearman… ranks Tranception above EVmutation in 15 of its 19 comparisons and EVmutation higher in 3; 1 is a tie.
ProteinGym AMFR assay-level Spearman comparison ranks Tranception above CARP in 19 of 19 comparisons; other studies rank CARP above Tranception in 6 of 6.
Livesey and Marsh 2025 ranks CARP above Tranception in all 6 of its comparisons. ProteinGym AMFR assay-level Spearman… ranks Tranception above CARP in all 19 of its comparisons.
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks UTR-LM above DNABERT in 23 of 25 comparisons; other studies rank DNABERT above UTR-LM in 8 of 8.
Benchmarking pre-trained genomic… ranks DNABERT above UTR-LM in all 8 comparisons that are not ties (4 are ties). Zero-shot benchmarking of RNA language… ranks UTR-LM above DNABERT in 23 of its 30 comparisons and DNABERT higher in 2; 5 are ties.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above BayesDel in 6 of 7 comparisons; other studies rank BayesDel above ESM-1b in 14 of 14.
2 papers rank BayesDel above ESM-1b in all 14 of their comparisons. Livesey and Marsh 2025 ranks ESM-1b above BayesDel in 6 of its 7 comparisons and BayesDel higher in 1.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MetaLR above LIST-S2 in 7 of 7 comparisons; other studies rank LIST-S2 above MetaLR in 4 of 5.
An openly licensed benchmark… ranks LIST-S2 above MetaLR in 4 of its 6 comparisons and MetaLR higher in 1; 1 is a tie. Livesey and Marsh 2025 ranks MetaLR above LIST-S2 in all 7 of its comparisons.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above PolyPhen-2 in 6 of 6 comparisons; other studies rank PolyPhen-2 above MutFormer in 6 of 6.
An openly licensed benchmark… ranks PolyPhen-2 above MutFormer in all 6 of its comparisons. Livesey and Marsh 2025 ranks MutFormer above PolyPhen-2 in all 6 of its comparisons.
An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks PROVEAN above MutFormer in 6 of 6 comparisons; other studies rank MutFormer above PROVEAN in 7 of 7.
Livesey and Marsh 2025 ranks MutFormer above PROVEAN in all 7 of its comparisons. An openly licensed benchmark… ranks PROVEAN above MutFormer in all 6 of its comparisons.
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above LucaOne in 6 of 6 comparisons; other studies rank LucaOne above Evo in 40 of 45.
GENEB leaderboard ranks LucaOne above Evo in 40 of its 45 comparisons and Evo higher in 5. Zero-shot benchmarking of RNA language… ranks Evo above LucaOne in all 6 of its comparisons.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above DEOGEN2 in 7 of 7 comparisons; other studies rank DEOGEN2 above MutFormer in 6 of 6.
An openly licensed benchmark… ranks DEOGEN2 above MutFormer in all 6 of its comparisons. Livesey and Marsh 2025 ranks MutFormer above DEOGEN2 in all 7 of its comparisons.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks M-CAP above PrimateAI in 7 of 7 comparisons; other studies rank PrimateAI above M-CAP in 24 of 30.
3 papers rank PrimateAI above M-CAP in 24 of their 30 comparisons and M-CAP higher in 6. Livesey and Marsh 2025 ranks M-CAP above PrimateAI in all 7 of its comparisons.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks popEVE above VARITY in 6 of 6 comparisons; other studies rank VARITY above popEVE in 4 of 5.
An openly licensed benchmark… ranks VARITY above popEVE in 4 of its 6 comparisons and popEVE higher in 1; 1 is a tie. Livesey and Marsh 2025 ranks popEVE above VARITY in all 6 comparisons that are not ties (1 is a tie).