rewirebio.iobenchmarks
Contested result

RNAErnie and Evo: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • Same task, different protocols

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above RNAErnie in 6 of 6 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks RNAErnie above Evo in 4 of 4 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank RNAErnie above Evo in 4 of their 10 comparisons and Evo higher in 6. Of those papers, Zero-shot benchmarking of RNA language… ranks Evo higher on its own (6 of 6).

Papers grouped by which system they rank higher: RNAErnie or EvoRank RNAErnie higher: RNAGym fitness leaderboard. Rank Evo higher: Zero-shot benchmarking of RNA language…. Mixed: none.Rank RNAErnie higherMixedRank Evo higherRNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdZero-shot benchmarkingof RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning
  • Rank RNAErnie higher (1): RNAGym fitness leaderboard
  • Rank Evo higher (1): Zero-shot benchmarking of RNA language…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores RNAErnie or Evo higherZero-shot benchmarking of RNA language…: RNAErnie higher in 0, Evo higher in 6, 0 ties. RNAGym fitness leaderboard: RNAErnie higher in 4, Evo higher in 0, 0 ties.Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning6 Evo higher6RNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md4 RNAErnie higher4
RNAErnie higherTieEvo higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Zero-shot benchmarking of RNA language… (6)
Comparisons in RNAGym fitness leaderboard (4)
ProtocolDatasetMetricRNAErnieEvoHigher
RNAGym v0.1.1 fitness, signed Spearman, AptamerRNAGym Aptamer DMS assays (2)Spearman (mean signed Spearman over the 2 Aptamer assays), unitless0.03060.0007RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesRNAGym ncRNA DMS assays (31)Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless0.1940.138RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, RibozymeRNAGym Ribozyme DMS assays (26)Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless0.1340.0278RNAErnie
RNAGym v0.1.1 fitness, signed Spearman, tRNARNAGym tRNA DMS assays (3)Spearman (mean signed Spearman over the 3 tRNA assays), unitless0.4160.385RNAErnie

What disagrees

studies
2
Compared
RNAErnie, Evo
Study that disagrees
Other studies
Rule
Two-study conflict; same task, different protocols
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-befd4edf81887f7d · All contested results · How findings are made and followed up