rewirebio.iobenchmarks
Contested result

Evo and AIDO.RNA: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • Same task, different protocols

Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning ranks Evo above AIDO.RNA in 5 of 5 comparisons; RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks AIDO.RNA above Evo in 3 of 3 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank Evo above AIDO.RNA in 5 of their 10 comparisons and AIDO.RNA higher in 3; 2 are ties. Of those papers, RNAGym fitness leaderboard ranks AIDO.RNA higher on its own (3 of 4).

Papers grouped by which system they rank higher: Evo or AIDO.RNARank Evo higher: Zero-shot benchmarking of RNA language…. Rank AIDO.RNA higher: RNAGym fitness leaderboard. Mixed: none.Rank Evo higherMixedRank AIDO.RNA higherZero-shot benchmarkingof RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learningRNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md
  • Rank Evo higher (1): Zero-shot benchmarking of RNA language…
  • Rank AIDO.RNA higher (1): RNAGym fitness leaderboard
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Evo or AIDO.RNA higherZero-shot benchmarking of RNA language…: Evo higher in 5, AIDO.RNA higher in 0, 1 tie. RNAGym fitness leaderboard: Evo higher in 0, AIDO.RNA higher in 3, 1 tie.Zero-shot benchmarking of RNA language…Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning5 Evo higher51 tie1RNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md1 tie13 AIDO.RNA higher3
Evo higherTieAIDO.RNA higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Zero-shot benchmarking of RNA language… (6)
Comparisons in RNAGym fitness leaderboard (4)
ProtocolDatasetMetricEvoAIDO.RNAHigher
RNAGym v0.1.1 fitness, signed Spearman, AptamerRNAGym Aptamer DMS assays (2)Spearman (mean signed Spearman over the 2 Aptamer assays), unitless0.00070.0847AIDO.RNA
RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesRNAGym ncRNA DMS assays (31)Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless0.1380.182AIDO.RNA
RNAGym v0.1.1 fitness, signed Spearman, RibozymeRNAGym Ribozyme DMS assays (26)Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless0.02780.074AIDO.RNA
RNAGym v0.1.1 fitness, signed Spearman, tRNARNAGym tRNA DMS assays (3)Spearman (mean signed Spearman over the 3 tRNA assays), unitless0.3850.389Tie

What disagrees

studies
2
Compared
Evo, AIDO.RNA
Study that disagrees
Other studies
Rule
Two-study conflict; same task, different protocols
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-d194c8b448707ff4 · All contested results · How findings are made and followed up