Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.
These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.
A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM3 above ESM-2 in 18 of 18 comparisons; pfmbench primary benchmark evidence ranks ESM-2 above ESM3 in 9 of 12 comparisons. No other study gives a verdict, so neither is the outlier.
3 papers rank ESM-2 above ESM3 in 20 of their 59 comparisons and ESM3 higher in 32; 7 are ties. Of those papers, ProteinGym ranks ESM3 higher on its own (18 of 19).
On ProteinGym mutation effects, ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design reports ESM3 at a median relative difference of +0.20 from ESM-2; other studies report -0.06 to -0.01.
In 2 papers, ESM3 scores between 1% and 6% below ESM-2. In ProteinGym, ESM3 scores 20% above ESM-2.
Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods ranks ESM-1b above EVE in 6 of 6 comparisons; 2 of 2 other studies rank EVE above ESM-1b.
2 papers rank EVE above ESM-1b in 22 of their 26 comparisons and ESM-1b higher in 1; 3 are ties. Brandes et al. 2023 ranks ESM-1b above EVE in all 6 comparisons that are not ties (2 are ties).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks VESPAl above CARP in 15 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above VESPAl in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank CARP above VESPAl in 10 of their 25 comparisons and VESPAl higher in 15. Of those papers, ProteinGym ranks VESPAl higher on its own (15 of 19).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above ESM-1b in 15 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1b above Tranception in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank ESM-1b above Tranception in 11 of their 26 comparisons and Tranception higher in 15. Of those papers, ProteinGym ranks Tranception higher on its own (15 of 19).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM-IF1 above ProteinMPNN in 19 of 19 comparisons; proteinbench primary benchmark evidence ranks ProteinMPNN above ESM-IF1 in 10 of 10 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank ESM-IF1 above ProteinMPNN in 19 of their 29 comparisons and ProteinMPNN higher in 10. Of those papers, proteinbench primary benchmark evidence ranks ProteinMPNN higher on its own (10 of 10).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks VESPA above CARP in 18 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above VESPA in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank CARP above VESPA in 7 of their 25 comparisons and VESPA higher in 18. Of those papers, ProteinGym ranks VESPA higher on its own (18 of 19).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above ESM-2 in 16 of 18 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-2 above Tranception in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank ESM-2 above Tranception in 9 of their 26 comparisons and Tranception higher in 16; 1 is a tie. Of those papers, ProteinGym ranks Tranception higher on its own (16 of 19).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks DeepSequence above ESM-1v in 13 of 17 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1v above DeepSequence in 6 of 7 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank ESM-1v above DeepSequence in 10 of their 26 comparisons and DeepSequence higher in 14; 2 are ties. Of those papers, ProteinGym ranks DeepSequence higher on its own (13 of 19).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM-1b above CARP in 14 of 18 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above ESM-1b in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank CARP above ESM-1b in 10 of their 25 comparisons and ESM-1b higher in 14; 1 is a tie. Of those papers, ProteinGym ranks ESM-1b higher on its own (14 of 19).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above CARP in 19 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above Tranception in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank CARP above Tranception in 6 of their 25 comparisons and Tranception higher in 19. Of those papers, ProteinGym ranks Tranception higher on its own (19 of 19).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks VESPA above DeepSequence in 13 of 17 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks DeepSequence above VESPA in 5 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank DeepSequence above VESPA in 9 of their 26 comparisons and VESPA higher in 14; 3 are ties. Of those papers, ProteinGym ranks VESPA higher on its own (13 of 19).
Transfer learning to leverage larger datasets for improved prediction of protein stability changes ranks MUpro above ThermoNet in 5 of 6 comparisons; Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset ranks ThermoNet above MUpro in 9 of 9 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank MUpro above ThermoNet in 5 of their 16 comparisons and ThermoNet higher in 10; 1 is a tie. Of those papers, Predicting protein stability changes… ranks ThermoNet higher on its own (9 of 10).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks MSA Transformer above ESM-1v in 16 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1v above MSA Transformer in 5 of 5 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank ESM-1v above MSA Transformer in 8 of their 25 comparisons and MSA Transformer higher in 16; 1 is a tie. Of those papers, ProteinGym ranks MSA Transformer higher on its own (16 of 19).
Transfer learning to leverage larger datasets for improved prediction of protein stability changes ranks MAESTRO above ACDC-NN in 5 of 5 comparisons; Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset ranks ACDC-NN above MAESTRO in 7 of 9 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank ACDC-NN above MAESTRO in 7 of their 16 comparisons and MAESTRO higher in 7; 2 are ties. Of those papers, Transfer learning to leverage larger… ranks MAESTRO higher on its own (5 of 6).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above EVmutation in 15 of 18 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks EVmutation above Tranception in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank EVmutation above Tranception in 10 of their 26 comparisons and Tranception higher in 15; 1 is a tie. Of those papers, ProteinGym ranks Tranception higher on its own (15 of 19).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above VESPAl in 17 of 18 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks VESPAl above Tranception in 6 of 7 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank Tranception above VESPAl in 18 of their 26 comparisons and VESPAl higher in 7; 1 is a tie. Of those papers, Livesey and Marsh 2025 ranks VESPAl higher on its own (6 of 7).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above ESM-1v in 14 of 17 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-1v above Tranception in 7 of 7 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank ESM-1v above Tranception in 10 of their 26 comparisons and Tranception higher in 14; 2 are ties. Of those papers, ProteinGym ranks Tranception higher on its own (14 of 19).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks Tranception above ProGen2 in 19 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ProGen2 above Tranception in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank ProGen2 above Tranception in 6 of their 26 comparisons and Tranception higher in 19; 1 is a tie. Of those papers, ProteinGym ranks Tranception higher on its own (19 of 20).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks EVmutation above CARP in 15 of 19 comparisons; Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks CARP above EVmutation in 5 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank EVmutation above CARP in 16 of their 25 comparisons and CARP higher in 9. Of those papers, Livesey and Marsh 2025 ranks CARP higher on its own (5 of 6).
ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design ranks ESM3 above ProteinMPNN in 18 of 19 comparisons; proteinbench primary benchmark evidence ranks ProteinMPNN above ESM3 in 10 of 10 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank ESM3 above ProteinMPNN in 18 of their 29 comparisons and ProteinMPNN higher in 11. Of those papers, proteinbench primary benchmark evidence ranks ProteinMPNN higher on its own (10 of 10).
Transfer learning to leverage larger datasets for improved prediction of protein stability changes ranks mCSM above ThermoNet in 6 of 6 comparisons; Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset ranks ThermoNet above mCSM in 8 of 10 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank mCSM above ThermoNet in 8 of their 16 comparisons and ThermoNet higher in 8. Of those papers, Predicting protein stability changes… ranks ThermoNet higher on its own (8 of 10).