Nucleotide Transformer and Evo: RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md reverses the order
- Low severity
- Not yet reviewed
- Ordering reversal
- different-task
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks Evo above Nucleotide Transformer in 3 of 4 comparisons; 2 of 2 other studies rank Nucleotide Transformer above Evo.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
3 papers rank Nucleotide Transformer above Evo in 72 of their 145 comparisons and Evo higher in 65; 8 are ties. Of those papers, Genome modelling and design across all… ranks Evo higher on its own (61 of 90). RNAGym fitness leaderboard ranks Evo above Nucleotide Transformer in 3 of its 4 comparisons and Nucleotide Transformer higher in 1.
- Rank Nucleotide Transformer higher (2): GENEB leaderboard, BacBench LICENSE file at commit 64b7d45
- Rank Evo higher (2): RNAGym fitness leaderboard (disagrees), Genome modelling and design across all…
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks Nucleotide Transformer higher | Ranks Evo higher | Ties |
|---|---|---|---|---|
| RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md · Repository | Disagrees | 1 | 3 | 0 |
| Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) and 1 more record · DOI | Agrees | 22 | 61 | 7 |
| GENEB leaderboard (leaderboard.json, 13 task-type aggregates) · Repository | Agrees | 40 | 4 | 1 |
| BacBench LICENSE file at commit 64b7d45 and 12 more records · DOI | Agrees | 10 | 0 | 0 |
Comparisons in RNAGym fitness leaderboard (4)
| Protocol | Dataset | Metric | Nucleotide Transformer | Evo | Higher |
|---|---|---|---|---|---|
| RNAGym v0.1.1 fitness, signed Spearman, Aptamer | RNAGym Aptamer DMS assays (2) | Spearman (mean signed Spearman over the 2 Aptamer assays), unitless | -0.0886 | 0.0007 | Evo |
| RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categories | RNAGym ncRNA DMS assays (31) | Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless | 0.12 | 0.138 | Evo |
| RNAGym v0.1.1 fitness, signed Spearman, Ribozyme | RNAGym Ribozyme DMS assays (26) | Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless | 0.133 | 0.0278 | Nucleotide Transformer |
| RNAGym v0.1.1 fitness, signed Spearman, tRNA | RNAGym tRNA DMS assays (3) | Spearman (mean signed Spearman over the 3 tRNA assays), unitless | 0.317 | 0.385 | Evo |
Comparisons in Genome modelling and design across all… (90)
Comparisons in GENEB leaderboard (45)
| Protocol | Dataset | Metric | Nucleotide Transformer | Evo | Higher |
|---|---|---|---|---|---|
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Chromatin Acc.'), unitless | 0.41 | 0.289 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Coding/NC'), unitless | 0.667 | 0.539 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'DNA Methyl.'), unitless | 0.0575 | 0.0458 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Enhancers'), unitless | 0.314 | 0.288 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Histone Mod.'), unitless | 0.258 | 0.258 | Tie |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Mouse Enh.'), unitless | 0.202 | 0.12 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Promoters'), unitless | 0.532 | 0.436 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Regulatory'), unitless | 0.152 | 0.125 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Species Clf.'), unitless | 0.512 | 0.207 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Splice Sites'), unitless | 0.242 | 0.0657 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'TF Binding'), unitless | 0.201 | 0.0386 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Virus/Phage'), unitless | 0.161 | 0.0941 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'lncRNA'), unitless | 0.103 | 0.0189 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; macro average: unweighted mean of the 13 category MCCs), unitless | 0.293 | 0.194 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; micro average: mean over all 100 tasks), unitless | 0.3 | 0.232 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Chromatin Acc.'), unitless | 0.169 | 0.14 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Coding/NC'), unitless | 0.251 | 0.293 | Evo |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'DNA Methyl.'), unitless | 0.0179 | 0.0208 | Evo |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Enhancers'), unitless | 0.171 | 0.165 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Histone Mod.'), unitless | 0.0952 | 0.108 | Evo |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Mouse Enh.'), unitless | 0.0481 | 0.0568 | Evo |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Promoters'), unitless | 0.309 | 0.279 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Regulatory'), unitless | 0.0837 | 0.0788 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Species Clf.'), unitless | 0.192 | 0.103 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Splice Sites'), unitless | 0.0768 | 0.0643 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'TF Binding'), unitless | 0.0361 | 0.0084 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Virus/Phage'), unitless | 0.0194 | -0.00343 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'lncRNA'), unitless | 0.0454 | 0.00964 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; macro average: unweighted mean of the 13 category MCCs), unitless | 0.116 | 0.102 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; micro average: mean over all 100 tasks), unitless | 0.136 | 0.126 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Chromatin Acc.'), unitless | 0.554 | 0.284 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Coding/NC'), unitless | 0.864 | 0.719 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'DNA Methyl.'), unitless | 0.356 | 0.0732 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Enhancers'), unitless | 0.487 | 0.37 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Histone Mod.'), unitless | 0.498 | 0.397 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Mouse Enh.'), unitless | 0.547 | 0.287 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Promoters'), unitless | 0.739 | 0.537 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Regulatory'), unitless | 0.488 | 0.269 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Species Clf.'), unitless | 0.752 | 0.285 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Splice Sites'), unitless | 0.652 | 0.16 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'TF Binding'), unitless | 0.573 | 0.173 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Virus/Phage'), unitless | 0.564 | 0.205 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'lncRNA'), unitless | 0.372 | 0.119 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; macro average: unweighted mean of the 13 category MCCs), unitless | 0.573 | 0.298 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; micro average: mean over all 100 tasks), unitless | 0.561 | 0.342 | Nucleotide Transformer |
Comparisons in BacBench LICENSE file at commit 64b7d45 (10)
| Protocol | Dataset | Metric | Nucleotide Transformer | Evo | Higher |
|---|---|---|---|---|---|
| BacBench antibiotic resistance, binary linear probe AUPRC | Antibiotic resistance binary prediction subset, 37 antibiotics | AUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless | 59.7 | 44.1 | Nucleotide Transformer |
| BacBench antibiotic resistance, binary linear probe AUROC | Antibiotic resistance binary prediction subset, 37 antibiotics | AUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless | 84.2 | 68.2 | Nucleotide Transformer |
| BacBench antibiotic resistance, MIC regression Pearson correlation | Antibiotic resistance MIC regression subset, 56 antibiotics | Pearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless | 51.2 | 29.2 | Nucleotide Transformer |
| BacBench antibiotic resistance, MIC regression R squared | Antibiotic resistance MIC regression subset, 56 antibiotics | R² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless | 27.7 | 10 | Nucleotide Transformer |
| BacBench essential gene prediction, linear probe AUPRC | BacBench essential gene dataset from the Database of Essential Genes | AUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless | 26 | 13.7 | Nucleotide Transformer |
| BacBench essential gene prediction, linear probe AUROC | BacBench essential gene dataset from the Database of Essential Genes | AUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless | 65.7 | 51.8 | Nucleotide Transformer |
| BacBench phenotypic trait prediction, linear probe AUPRC | BacBench phenotypic trait dataset | AUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless | 43 | 39.5 | Nucleotide Transformer |
| BacBench phenotypic trait prediction, linear probe AUROC | BacBench phenotypic trait dataset | AUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless | 66 | 60.2 | Nucleotide Transformer |
| BacBench protein-protein interaction, linear probe AUPRC | BacBench protein-protein interaction dataset from STRING DB | AUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless | 31.7 | 20.9 | Nucleotide Transformer |
| BacBench protein-protein interaction, linear probe AUROC | BacBench protein-protein interaction dataset from STRING DB | AUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless | 63.9 | 56.5 | Nucleotide Transformer |
What disagrees
- Other studies
- rank Nucleotide Transformer above Evo in 50 of 54 comparisons
- This study
- ranks Evo above Nucleotide Transformer in 3 of 4 comparisons
- Compared
- Nucleotide Transformer, Evo
- Study that disagrees
- RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md
- Other studies
14 studies
BacBench LICENSE file at commit 64b7d45, BacBench repository README at commit 64b7d45, Bacformer LICENSE file at commit 61dca95, Bacformer repository README at commit 61dca95, Hugging Face model card for macwiatrak/bacformer-causal-complete-genomes, Hugging Face model card for macwiatrak/bacformer-causal-MAG, Hugging Face model card for macwiatrak/bacformer-for-ppi-prediction, Hugging Face model card for macwiatrak/bacformer-large-masked-complete-genomes, Hugging Face model card for macwiatrak/bacformer-large-masked-MAG, Hugging Face model card for macwiatrak/bacformer-masked-complete-genomes, Hugging Face model card for macwiatrak/bacformer-masked-MAG, Bacformer preprint, bioRxiv 2025.07.20.665723 version 2, Bacformer preprint, bioRxiv 2025.07.20.665723 version 3, GENEB leaderboard (leaderboard.json, 13 task-type aggregates)- Rule
- Ordering reversal; different-task
- Model identity
- Matched across studies partly through links not yet reviewed
The compared numbers
Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.
In the other studies (108 results)
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-e05599ffec82fd2f · All contested results · How findings are made and followed up