Evo and AIDO.RNA: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md disagree
- Low severity
- Not yet reviewed
- Two-study conflict
- Same task, different protocols
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above AIDO.RNA in 5 of 5 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks AIDO.RNA above Evo in 3 of 3 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
2 papers rank Evo above AIDO.RNA in 5 of their 10 comparisons and AIDO.RNA higher in 3; 2 are ties. Of those papers, RNAGym fitness leaderboard ranks AIDO.RNA higher on its own (3 of 4).
- Rank Evo higher (1): Zero-shot benchmarking of RNA language…
- Rank AIDO.RNA higher (1): RNAGym fitness leaderboard
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks Evo higher | Ranks AIDO.RNA higher | Ties |
|---|---|---|---|---|
| Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOI | Agrees | 5 | 0 | 1 |
| RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md · Repository | Agrees | 0 | 3 | 1 |
Comparisons in Zero-shot benchmarking of RNA language… (6)
Comparisons in RNAGym fitness leaderboard (4)
| Protocol | Dataset | Metric | Evo | AIDO.RNA | Higher |
|---|---|---|---|---|---|
| RNAGym v0.1.1 fitness, signed Spearman, Aptamer | RNAGym Aptamer DMS assays (2) | Spearman (mean signed Spearman over the 2 Aptamer assays), unitless | 0.0007 | 0.0847 | AIDO.RNA |
| RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categories | RNAGym ncRNA DMS assays (31) | Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless | 0.138 | 0.182 | AIDO.RNA |
| RNAGym v0.1.1 fitness, signed Spearman, Ribozyme | RNAGym Ribozyme DMS assays (26) | Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless | 0.0278 | 0.074 | AIDO.RNA |
| RNAGym v0.1.1 fitness, signed Spearman, tRNA | RNAGym tRNA DMS assays (3) | Spearman (mean signed Spearman over the 3 tRNA assays), unitless | 0.385 | 0.389 | Tie |
What disagrees
- studies
- 2
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-d194c8b448707ff4 · All contested results · How findings are made and followed up