RNAErnie and Evo: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md disagree
- Low severity
- Not yet reviewed
- Two-study conflict
- Same task, different protocols
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above RNAErnie in 6 of 6 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above Evo in 4 of 4 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
2 papers rank RNAErnie above Evo in 4 of their 10 comparisons and Evo higher in 6. Of those papers, Zero-shot benchmarking of RNA language… ranks Evo higher on its own (6 of 6).
- Rank RNAErnie higher (1): RNAGym fitness leaderboard
- Rank Evo higher (1): Zero-shot benchmarking of RNA language…
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks RNAErnie higher | Ranks Evo higher | Ties |
|---|---|---|---|---|
| Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and 1 more record · DOI | Agrees | 0 | 6 | 0 |
| RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md · Repository | Agrees | 4 | 0 | 0 |
Comparisons in Zero-shot benchmarking of RNA language… (6)
Comparisons in RNAGym fitness leaderboard (4)
| Protocol | Dataset | Metric | RNAErnie | Evo | Higher |
|---|---|---|---|---|---|
| RNAGym v0.1.1 fitness, signed Spearman, Aptamer | RNAGym Aptamer DMS assays (2) | Spearman (mean signed Spearman over the 2 Aptamer assays), unitless | 0.0306 | 0.0007 | RNAErnie |
| RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categories | RNAGym ncRNA DMS assays (31) | Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless | 0.194 | 0.138 | RNAErnie |
| RNAGym v0.1.1 fitness, signed Spearman, Ribozyme | RNAGym Ribozyme DMS assays (26) | Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless | 0.134 | 0.0278 | RNAErnie |
| RNAGym v0.1.1 fitness, signed Spearman, tRNA | RNAGym tRNA DMS assays (3) | Spearman (mean signed Spearman over the 3 tRNA assays), unitless | 0.416 | 0.385 | RNAErnie |
What disagrees
- studies
- 2
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-befd4edf81887f7d · All contested results · How findings are made and followed up