Contested result
Nucleotide Transformer v2 and Evo: GENEB leaderboard (leaderboard.json, 13 task-type aggregates) reverses the order
- Low severity
- Not yet reviewed
- Ordering reversal
- Different tasks across studies
GENEB leaderboard (leaderboard.json, 13 task-type aggregates) ranks Nucleotide Transformer v2 above Evo in 42 of 45 comparisons; other studies rank Evo above Nucleotide Transformer v2 in 6 of 6.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
Zero-shot benchmarking of RNA language… ranks Evo above Nucleotide Transformer v2 in all 6 of its comparisons. GENEB leaderboard ranks Nucleotide Transformer v2 above Evo in 42 of its 45 comparisons and Evo higher in 3.
- Rank Evo higher (1): Zero-shot benchmarking of RNA language…
- Rank Nucleotide Transformer v2 higher (1): GENEB leaderboard (disagrees)
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks Evo higher | Ranks Nucleotide Transformer v2 higher | Ties |
|---|---|---|---|---|
| GENEB leaderboard (leaderboard.json, 13 task-type aggregates) · Repository | Disagrees | 3 | 42 | 0 |
| Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOI | Agrees | 6 | 0 | 0 |
Comparisons in GENEB leaderboard (45)
| Protocol | Dataset | Metric | Evo | Nucleotide Transformer v2 | Higher |
|---|---|---|---|---|---|
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Chromatin Acc.'), unitless | 0.289 | 0.481 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Coding/NC'), unitless | 0.539 | 0.587 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'DNA Methyl.'), unitless | 0.0458 | 0.0804 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Enhancers'), unitless | 0.288 | 0.3 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Histone Mod.'), unitless | 0.258 | 0.261 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Mouse Enh.'), unitless | 0.12 | 0.171 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Promoters'), unitless | 0.436 | 0.545 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Regulatory'), unitless | 0.125 | 0.182 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Species Clf.'), unitless | 0.207 | 0.564 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Splice Sites'), unitless | 0.0657 | 0.129 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'TF Binding'), unitless | 0.0386 | 0.242 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Virus/Phage'), unitless | 0.0941 | 0.147 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'lncRNA'), unitless | 0.0189 | 0.0528 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; macro average: unweighted mean of the 13 category MCCs), unitless | 0.194 | 0.286 | Nucleotide Transformer v2 |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; micro average: mean over all 100 tasks), unitless | 0.232 | 0.293 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Chromatin Acc.'), unitless | 0.14 | 0.186 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Coding/NC'), unitless | 0.293 | 0.284 | Evo |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'DNA Methyl.'), unitless | 0.0208 | 0.0223 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Enhancers'), unitless | 0.165 | 0.159 | Evo |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Histone Mod.'), unitless | 0.108 | 0.115 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Mouse Enh.'), unitless | 0.0568 | 0.0424 | Evo |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Promoters'), unitless | 0.279 | 0.36 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Regulatory'), unitless | 0.0788 | 0.113 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Species Clf.'), unitless | 0.103 | 0.304 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Splice Sites'), unitless | 0.0643 | 0.0704 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'TF Binding'), unitless | 0.0084 | 0.0992 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Virus/Phage'), unitless | -0.00343 | 0.0356 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'lncRNA'), unitless | 0.00964 | 0.037 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; macro average: unweighted mean of the 13 category MCCs), unitless | 0.102 | 0.136 | Nucleotide Transformer v2 |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; micro average: mean over all 100 tasks), unitless | 0.126 | 0.158 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Chromatin Acc.'), unitless | 0.284 | 0.652 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Coding/NC'), unitless | 0.719 | 0.812 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'DNA Methyl.'), unitless | 0.0732 | 0.339 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Enhancers'), unitless | 0.37 | 0.469 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Histone Mod.'), unitless | 0.397 | 0.473 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Mouse Enh.'), unitless | 0.287 | 0.492 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Promoters'), unitless | 0.537 | 0.722 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Regulatory'), unitless | 0.269 | 0.457 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Species Clf.'), unitless | 0.285 | 0.747 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Splice Sites'), unitless | 0.16 | 0.47 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'TF Binding'), unitless | 0.173 | 0.55 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Virus/Phage'), unitless | 0.205 | 0.482 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'lncRNA'), unitless | 0.119 | 0.337 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; macro average: unweighted mean of the 13 category MCCs), unitless | 0.298 | 0.537 | Nucleotide Transformer v2 |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; micro average: mean over all 100 tasks), unitless | 0.342 | 0.524 | Nucleotide Transformer v2 |
Comparisons in Zero-shot benchmarking of RNA language… (6)
What disagrees
- Other studies
- rank Evo above Nucleotide Transformer v2 in 6 of 6 comparisons
- This study
- ranks Nucleotide Transformer v2 above Evo in 42 of 45 comparisons
- Compared
- Nucleotide Transformer v2, Evo
- Study that disagrees
- GENEB leaderboard (leaderboard.json, 13 task-type aggregates)
- Other studies
- Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning, Zero-shot benchmarking of RNA language models: Supplementary Information
- Rule
- Ordering reversal; different tasks across studies
- Model identity
- Matched across studies partly through links not yet reviewed
The compared numbers
Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-49945cc909addf85 · All contested results · How findings are made and followed up