Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.
These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.
A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks MVP above ESM-1b in 2 of 2 comparisons; 2 of 2 other studies rank ESM-1b above MVP.
2 papers rank ESM-1b above MVP in all 13 of their comparisons. Brandes et al. 2023 ranks MVP above ESM-1b in all 2 comparisons that are not ties (2 are ties).
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above BayesDel in 6 of 7 comparisons; 2 of 2 other studies rank BayesDel above ESM-1b.
2 papers rank BayesDel above ESM-1b in all 10 of their comparisons. Livesey and Marsh 2025 ranks ESM-1b above BayesDel in 6 of its 7 comparisons and BayesDel higher in 1.
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks MetaLR above ESM-1b in 2 of 2 comparisons; 2 of 2 other studies rank ESM-1b above MetaLR.
2 papers rank ESM-1b above MetaLR in all 13 of their comparisons. Brandes et al. 2023 ranks MetaLR above ESM-1b in all 2 comparisons that are not ties (2 are ties).
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks MetaSVM above ESM-1b in 2 of 2 comparisons; 2 of 2 other studies rank ESM-1b above MetaSVM.
2 papers rank ESM-1b above MetaSVM in all 13 of their comparisons. Brandes et al. 2023 ranks MetaSVM above ESM-1b in all 2 comparisons that are not ties (2 are ties).
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above MutPred in 6 of 7 comparisons; 2 of 2 other studies rank MutPred above ESM-1b.
3 papers rank MutPred above ESM-1b in 8 of their 11 comparisons and ESM-1b higher in 1; 2 are ties. Of those papers, ProteinGym ranks ESM-1b higher on its own (1 of 1). Livesey and Marsh 2025 ranks ESM-1b above MutPred in 6 of its 7 comparisons and MutPred higher in 1.
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above VEST4 in 6 of 7 comparisons; 2 of 2 other studies rank VEST4 above ESM-1b.
2 papers rank VEST4 above ESM-1b in 7 of their 10 comparisons and ESM-1b higher in 1; 2 are ties. Livesey and Marsh 2025 ranks ESM-1b above VEST4 in 6 of its 7 comparisons and VEST4 higher in 1.
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above ESM-1b in 15 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above Tranception in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank ESM-1b above Tranception in 11 of their 26 comparisons and Tranception higher in 15. Of those papers, ProteinGym ranks Tranception higher on its own (15 of 19).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM-1b above CARP in 14 of 18 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above ESM-1b in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank CARP above ESM-1b in 10 of their 25 comparisons and ESM-1b higher in 14; 1 is a tie. Of those papers, ProteinGym ranks ESM-1b higher on its own (14 of 19).
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks MutFormer above ESM-1b in 7 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks ESM-1b above MutFormer in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank MutFormer above ESM-1b in 7 of their 13 comparisons and ESM-1b higher in 6. Of those papers, An openly licensed benchmark… ranks ESM-1b higher on its own (6 of 6).
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks ESM-1b above MutationTaster in 4 of 4 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks MutationTaster above ESM-1b in 5 of 5 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank MutationTaster above ESM-1b in 5 of their 10 comparisons and ESM-1b higher in 4; 1 is a tie. Of those papers, Brandes et al. 2023 ranks ESM-1b higher on its own (4 of 4).
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above MutScore in 6 of 7 comparisons; An openly licensed benchmark and per-gene calibration map for missense pathogenicity predictors on activating cancer drivers ranks MutScore above ESM-1b in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank MutScore above ESM-1b in 7 of their 13 comparisons and ESM-1b higher in 6. Of those papers, Livesey and Marsh 2025 ranks ESM-1b higher on its own (6 of 7).