Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.
These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.
A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.
BacBench LICENSE file at commit 64b7d45 ranks Evo above Evo 2 in 8 of 10 comparisons; 2 of 2 other studies rank Evo 2 above Evo.
2 papers rank Evo 2 above Evo in all 94 of their comparisons. BacBench LICENSE file at commit 64b7d45 ranks Evo above Evo 2 in 8 of its 10 comparisons and Evo 2 higher in 2.
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks Evo above Nucleotide Transformer in 3 of 4 comparisons; 2 of 2 other studies rank Nucleotide Transformer above Evo.
3 papers rank Nucleotide Transformer above Evo in 72 of their 145 comparisons and Evo higher in 65; 8 are ties. Of those papers, Genome modelling and design across all… ranks Evo higher on its own (61 of 90). RNAGym fitness leaderboard ranks Evo above Nucleotide Transformer in 3 of its 4 comparisons and Nucleotide Transformer higher in 1.
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above DNABERT-2 in 6 of 6 comparisons; 2 of 2 other studies rank DNABERT-2 above Evo.
2 papers rank DNABERT-2 above Evo in 45 of their 55 comparisons and Evo higher in 10. Zero-shot benchmarking of RNA language… ranks Evo above DNABERT-2 in all 6 of its comparisons.
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above LucaOne in 6 of 6 comparisons; GENEB leaderboard (leaderboard.json, 13 task-type aggregates) ranks LucaOne above Evo in 40 of 45 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank Evo above LucaOne in 11 of their 51 comparisons and LucaOne higher in 40. Of those papers, GENEB leaderboard ranks LucaOne higher on its own (40 of 45).
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above RNAErnie in 6 of 6 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above Evo in 4 of 4 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank RNAErnie above Evo in 4 of their 10 comparisons and Evo higher in 6. Of those papers, Zero-shot benchmarking of RNA language… ranks Evo higher on its own (6 of 6).
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above AIDO.RNA in 5 of 5 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks AIDO.RNA above Evo in 3 of 3 comparisons. No other study gives a verdict, so neither is the outlier.
2 papers rank Evo above AIDO.RNA in 5 of their 10 comparisons and AIDO.RNA higher in 3; 2 are ties. Of those papers, RNAGym fitness leaderboard ranks AIDO.RNA higher on its own (3 of 4).