rewirebio.iobenchmarks
Contested result

Nucleotide Transformer and Evo: RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • different-task

RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md ranks Evo above Nucleotide Transformer in 3 of 4 comparisons; 2 of 2 other studies rank Nucleotide Transformer above Evo.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

3 papers rank Nucleotide Transformer above Evo in 72 of their 145 comparisons and Evo higher in 65; 8 are ties. Of those papers, Genome modelling and design across all… ranks Evo higher on its own (61 of 90). RNAGym fitness leaderboard ranks Evo above Nucleotide Transformer in 3 of its 4 comparisons and Nucleotide Transformer higher in 1.

Papers grouped by which system they rank higher: Nucleotide Transformer or EvoRank Nucleotide Transformer higher: GENEB leaderboard, BacBench LICENSE file at commit 64b7d45. Rank Evo higher: RNAGym fitness leaderboard (disagrees), Genome modelling and design across all…. Mixed: none.Rank Nucleotide Transformer h…MixedRank Evo higherGENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)BacBench LICENSE fileat commit 64b7d45BacBench LICENSE file at commit 64b7d45RNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdGenome modelling anddesign across all…Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026)
  • Rank Nucleotide Transformer higher (2): GENEB leaderboard, BacBench LICENSE file at commit 64b7d45
  • Rank Evo higher (2): RNAGym fitness leaderboard (disagrees), Genome modelling and design across all…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Nucleotide Transformer or Evo higherRNAGym fitness leaderboard (disagrees): Nucleotide Transformer higher in 1, Evo higher in 3, 0 ties. Genome modelling and design across all…: Nucleotide Transformer higher in 22, Evo higher in 61, 7 ties. GENEB leaderboard: Nucleotide Transformer higher in 40, Evo higher in 4, 1 tie. BacBench LICENSE file at commit 64b7d45: Nucleotide Transformer higher in 10, Evo higher in 0, 0 ties.Disagrees: RNAGym fitness leaderboardRNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md1 Nucleotide Transformer higher13 Evo higher3Genome modelling and design across all…Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026)22 Nucleotide Transformer higher227 tie61 Evo higher61GENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)40 Nucleotide Transformer higher401 tie4 Evo higherBacBench LICENSE file at commit 64b7d45BacBench LICENSE file at commit 64b7d4510 Nucleotide Transformer higher10
Nucleotide Transformer higherTieEvo higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in RNAGym fitness leaderboard (4)
ProtocolDatasetMetricNucleotide TransformerEvoHigher
RNAGym v0.1.1 fitness, signed Spearman, AptamerRNAGym Aptamer DMS assays (2)Spearman (mean signed Spearman over the 2 Aptamer assays), unitless-0.08860.0007Evo
RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesRNAGym ncRNA DMS assays (31)Spearman (macro-mean over 3 ncRNA categories (equal category weights)), unitless0.120.138Evo
RNAGym v0.1.1 fitness, signed Spearman, RibozymeRNAGym Ribozyme DMS assays (26)Spearman (mean signed Spearman over the 26 Ribozyme assays), unitless0.1330.0278Nucleotide Transformer
RNAGym v0.1.1 fitness, signed Spearman, tRNARNAGym tRNA DMS assays (3)Spearman (mean signed Spearman over the 3 tRNA assays), unitless0.3170.385Evo
Comparisons in Genome modelling and design across all… (90)
ProtocolDatasetMetricNucleotide TransformerEvoHigher
Zero-shot BRCA1 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUPRC (zero-shot; BRCA1 'Both' (coding and noncoding) SNVs), unitless0.2850.216Nucleotide Transformer
Zero-shot BRCA1 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUROC (zero-shot; BRCA1 'Both' (coding and noncoding) SNVs), unitless0.550.497Nucleotide Transformer
Zero-shot BRCA1 coding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUPRC (zero-shot; BRCA1 coding SNVs), unitless0.2120.231Evo
Zero-shot BRCA1 coding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUROC (zero-shot; BRCA1 coding SNVs), unitless0.5090.519Evo
Zero-shot BRCA1 intronic SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUPRC (zero-shot; BRCA1 intronic SNVs), unitless0.07250.0245Nucleotide Transformer
Zero-shot BRCA1 intronic SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUROC (zero-shot; BRCA1 intronic SNVs), unitless0.6880.465Nucleotide Transformer
Zero-shot BRCA1 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUPRC (zero-shot; BRCA1 noncoding SNVs), unitless0.4510.212Nucleotide Transformer
Zero-shot BRCA1 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUROC (zero-shot; BRCA1 noncoding SNVs), unitless0.6490.467Nucleotide Transformer
Zero-shot BRCA1 splice region SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUPRC (zero-shot; BRCA1 splice region SNVs), unitless0.6110.403Nucleotide Transformer
Zero-shot BRCA1 splice region SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUROC (zero-shot; BRCA1 splice region SNVs), unitless0.6340.462Nucleotide Transformer
Zero-shot BRCA2 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUPRC (zero-shot; BRCA2 'Both' (coding and noncoding) SNVs), unitless0.1760.228Evo
Zero-shot BRCA2 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUROC (zero-shot; BRCA2 'Both' (coding and noncoding) SNVs), unitless0.5140.54Evo
Zero-shot BRCA2 coding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUPRC (zero-shot; BRCA2 coding SNVs), unitless0.1340.157Evo
Zero-shot BRCA2 coding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUROC (zero-shot; BRCA2 coding SNVs), unitless0.5010.535Evo
Zero-shot BRCA2 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUPRC (zero-shot; BRCA2 noncoding SNVs), unitless0.3610.265Nucleotide Transformer
Zero-shot BRCA2 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUROC (zero-shot; BRCA2 noncoding SNVs), unitless0.5550.545Nucleotide Transformer
Zero-shot ClinVar 'Both' (coding and noncoding) indels classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar 'Both' (coding and noncoding) indels), unitless0.8520.918Evo
Zero-shot ClinVar 'Both' (coding and noncoding) indels classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar 'Both' (coding and noncoding) indels), unitless0.5050.655Evo
Zero-shot ClinVar 'Both' (coding and noncoding) microsatellite variants classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar 'Both' (coding and noncoding) microsatellite variants), unitless0.4610.925Evo
Zero-shot ClinVar 'Both' (coding and noncoding) microsatellite variants classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar 'Both' (coding and noncoding) microsatellite variants), unitless0.530.912Evo
Zero-shot ClinVar 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar 'Both' (coding and noncoding) SNVs), unitless0.4260.577Evo
Zero-shot ClinVar 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar 'Both' (coding and noncoding) SNVs), unitless0.5060.607Evo
Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding indels), unitless0.7010.803Evo
Zero-shot ClinVar coding indels classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding indels), unitless0.5030.691Evo
Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding microsatellite variants), unitless0.1710.773Evo
Zero-shot ClinVar coding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding microsatellite variants), unitless0.5350.795Evo
Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, far from splice site), unitless0.6710.778Evo
Zero-shot ClinVar coding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, far from splice site), unitless0.5070.687Evo
Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, highly conserved), unitless0.9370.943Tie
Zero-shot ClinVar coding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, highly conserved), unitless0.5210.526Tie
Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, moderately conserved), unitless0.6940.706Evo
Zero-shot ClinVar coding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, moderately conserved), unitless0.5170.532Evo
Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, not conserved), unitless0.5770.564Nucleotide Transformer
Zero-shot ClinVar coding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, not conserved), unitless0.520.529Evo
Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding non-SNVs, poorly conserved), unitless0.5680.546Nucleotide Transformer
Zero-shot ClinVar coding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding non-SNVs, poorly conserved), unitless0.5010.547Evo
Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs), unitless0.6590.732Evo
Zero-shot ClinVar coding SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs), unitless0.5050.56Evo
Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, far from splice site), unitless0.6490.726Evo
Zero-shot ClinVar coding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, far from splice site), unitless0.5060.56Evo
Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, highly conserved), unitless0.7970.847Evo
Zero-shot ClinVar coding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, highly conserved), unitless0.5010.558Evo
Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, moderately conserved), unitless0.3230.356Evo
Zero-shot ClinVar coding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, moderately conserved), unitless0.5140.503Nucleotide Transformer
Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, near splice site), unitless0.7770.837Evo
Zero-shot ClinVar coding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, near splice site), unitless0.5090.586Evo
Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, not conserved), unitless0.1440.185Evo
Zero-shot ClinVar coding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, not conserved), unitless0.5450.552Evo
Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar coding SNVs, poorly conserved), unitless0.1930.192Tie
Zero-shot ClinVar coding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar coding SNVs, poorly conserved), unitless0.5260.531Tie
Zero-shot ClinVar low SpliceAI and Pangolin scores (both below 0.1) non-SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar low SpliceAI and Pangolin scores (both below 0.1) non-SNVs), unitless0.01570.0323Evo
Zero-shot ClinVar low SpliceAI and Pangolin scores (both below 0.1) non-SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar low SpliceAI and Pangolin scores (both below 0.1) non-SNVs), unitless0.510.545Evo
Zero-shot ClinVar low SpliceAI and Pangolin scores (both below 0.1) SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar low SpliceAI and Pangolin scores (both below 0.1) SNVs), unitless0.005450.0162Evo
Zero-shot ClinVar low SpliceAI and Pangolin scores (both below 0.1) SNVs classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar low SpliceAI and Pangolin scores (both below 0.1) SNVs), unitless0.520.548Evo
Zero-shot ClinVar noncoding indels classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding indels), unitless0.1770.186Evo
Zero-shot ClinVar noncoding indels classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding indels), unitless0.5050.531Evo
Zero-shot ClinVar noncoding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding microsatellite variants), unitless0.03640.183Evo
Zero-shot ClinVar noncoding microsatellite variants classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding microsatellite variants), unitless0.7860.964Evo
Zero-shot ClinVar noncoding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding non-SNVs, far from splice site), unitless0.04430.0505Evo
Zero-shot ClinVar noncoding non-SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding non-SNVs, far from splice site), unitless0.5180.534Evo
Zero-shot ClinVar noncoding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding non-SNVs, highly conserved), unitless0.7320.778Evo
Zero-shot ClinVar noncoding non-SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding non-SNVs, highly conserved), unitless0.5030.569Evo
Zero-shot ClinVar noncoding non-SNVs, intergenic classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding non-SNVs, intergenic), unitless0.7290.708Nucleotide Transformer
Zero-shot ClinVar noncoding non-SNVs, intergenic classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding non-SNVs, intergenic), unitless0.6250.625Tie
Zero-shot ClinVar noncoding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding non-SNVs, moderately conserved), unitless0.1640.165Tie
Zero-shot ClinVar noncoding non-SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding non-SNVs, moderately conserved), unitless0.5020.508Evo
Zero-shot ClinVar noncoding non-SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding non-SNVs, near splice site), unitless0.50.544Evo
Zero-shot ClinVar noncoding non-SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding non-SNVs, near splice site), unitless0.5190.549Evo
Zero-shot ClinVar noncoding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding non-SNVs, not conserved), unitless0.04710.0278Nucleotide Transformer
Zero-shot ClinVar noncoding non-SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding non-SNVs, not conserved), unitless0.5840.631Evo
Zero-shot ClinVar noncoding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding non-SNVs, poorly conserved), unitless0.02430.0258Evo
Zero-shot ClinVar noncoding non-SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding non-SNVs, poorly conserved), unitless0.5430.506Nucleotide Transformer
Zero-shot ClinVar noncoding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, far from splice site), unitless0.01450.02Evo
Zero-shot ClinVar noncoding SNVs, far from splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, far from splice site), unitless0.510.528Evo
Zero-shot ClinVar noncoding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, highly conserved), unitless0.920.897Nucleotide Transformer
Zero-shot ClinVar noncoding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, highly conserved), unitless0.5520.531Nucleotide Transformer
Zero-shot ClinVar noncoding SNVs, intergenic classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, intergenic), unitless0.2160.293Evo
Zero-shot ClinVar noncoding SNVs, intergenic classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, intergenic), unitless0.6550.525Nucleotide Transformer
Zero-shot ClinVar noncoding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, moderately conserved), unitless0.07730.072Nucleotide Transformer
Zero-shot ClinVar noncoding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, moderately conserved), unitless0.5230.5Nucleotide Transformer
Zero-shot ClinVar noncoding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, near splice site), unitless0.8410.834Tie
Zero-shot ClinVar noncoding SNVs, near splice site classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, near splice site), unitless0.5330.551Evo
Zero-shot ClinVar noncoding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, not conserved), unitless0.0110.0116Evo
Zero-shot ClinVar noncoding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, not conserved), unitless0.5160.547Evo
Zero-shot ClinVar noncoding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, poorly conserved), unitless0.0230.0216Nucleotide Transformer
Zero-shot ClinVar noncoding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, poorly conserved), unitless0.5020.527Evo
Zero-shot SpliceVarDB 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)SpliceVarDB experimentally validated splice variants (Brixi et al. 2026)AUPRC (zero-shot; SpliceVarDB 'Both' (coding and noncoding) SNVs), unitless0.6010.621Evo
Zero-shot SpliceVarDB 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)SpliceVarDB experimentally validated splice variants (Brixi et al. 2026)AUROC (zero-shot; SpliceVarDB 'Both' (coding and noncoding) SNVs), unitless0.4910.534Evo
Zero-shot SpliceVarDB exonic SNVs classification (Evo 2 paper, Supplementary Data 1)SpliceVarDB experimentally validated splice variants (Brixi et al. 2026)AUPRC (zero-shot; SpliceVarDB exonic SNVs), unitless0.3850.414Evo
Zero-shot SpliceVarDB exonic SNVs classification (Evo 2 paper, Supplementary Data 1)SpliceVarDB experimentally validated splice variants (Brixi et al. 2026)AUROC (zero-shot; SpliceVarDB exonic SNVs), unitless0.490.558Evo
Comparisons in GENEB leaderboard (45)
ProtocolDatasetMetricNucleotide TransformerEvoHigher
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Chromatin Acc.'), unitless0.410.289Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Coding/NC'), unitless0.6670.539Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'DNA Methyl.'), unitless0.05750.0458Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Enhancers'), unitless0.3140.288Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Histone Mod.'), unitless0.2580.258Tie
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Mouse Enh.'), unitless0.2020.12Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Promoters'), unitless0.5320.436Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Regulatory'), unitless0.1520.125Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Species Clf.'), unitless0.5120.207Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Splice Sites'), unitless0.2420.0657Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'TF Binding'), unitless0.2010.0386Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Virus/Phage'), unitless0.1610.0941Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'lncRNA'), unitless0.1030.0189Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; macro average: unweighted mean of the 13 category MCCs), unitless0.2930.194Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; micro average: mean over all 100 tasks), unitless0.30.232Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Chromatin Acc.'), unitless0.1690.14Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Coding/NC'), unitless0.2510.293Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'DNA Methyl.'), unitless0.01790.0208Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Enhancers'), unitless0.1710.165Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Histone Mod.'), unitless0.09520.108Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Mouse Enh.'), unitless0.04810.0568Evo
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Promoters'), unitless0.3090.279Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Regulatory'), unitless0.08370.0788Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Species Clf.'), unitless0.1920.103Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Splice Sites'), unitless0.07680.0643Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'TF Binding'), unitless0.03610.0084Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Virus/Phage'), unitless0.0194-0.00343Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'lncRNA'), unitless0.04540.00964Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; macro average: unweighted mean of the 13 category MCCs), unitless0.1160.102Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; micro average: mean over all 100 tasks), unitless0.1360.126Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Chromatin Acc.'), unitless0.5540.284Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Coding/NC'), unitless0.8640.719Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'DNA Methyl.'), unitless0.3560.0732Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Enhancers'), unitless0.4870.37Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Histone Mod.'), unitless0.4980.397Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Mouse Enh.'), unitless0.5470.287Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Promoters'), unitless0.7390.537Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Regulatory'), unitless0.4880.269Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Species Clf.'), unitless0.7520.285Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Splice Sites'), unitless0.6520.16Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'TF Binding'), unitless0.5730.173Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Virus/Phage'), unitless0.5640.205Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'lncRNA'), unitless0.3720.119Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; macro average: unweighted mean of the 13 category MCCs), unitless0.5730.298Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; micro average: mean over all 100 tasks), unitless0.5610.342Nucleotide Transformer
Comparisons in BacBench LICENSE file at commit 64b7d45 (10)
ProtocolDatasetMetricNucleotide TransformerEvoHigher
BacBench antibiotic resistance, binary linear probe AUPRCAntibiotic resistance binary prediction subset, 37 antibioticsAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless59.744.1Nucleotide Transformer
BacBench antibiotic resistance, binary linear probe AUROCAntibiotic resistance binary prediction subset, 37 antibioticsAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless84.268.2Nucleotide Transformer
BacBench antibiotic resistance, MIC regression Pearson correlationAntibiotic resistance MIC regression subset, 56 antibioticsPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless51.229.2Nucleotide Transformer
BacBench antibiotic resistance, MIC regression R squaredAntibiotic resistance MIC regression subset, 56 antibioticsR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless27.710Nucleotide Transformer
BacBench essential gene prediction, linear probe AUPRCBacBench essential gene dataset from the Database of Essential GenesAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless2613.7Nucleotide Transformer
BacBench essential gene prediction, linear probe AUROCBacBench essential gene dataset from the Database of Essential GenesAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless65.751.8Nucleotide Transformer
BacBench phenotypic trait prediction, linear probe AUPRCBacBench phenotypic trait datasetAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless4339.5Nucleotide Transformer
BacBench phenotypic trait prediction, linear probe AUROCBacBench phenotypic trait datasetAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless6660.2Nucleotide Transformer
BacBench protein-protein interaction, linear probe AUPRCBacBench protein-protein interaction dataset from STRING DBAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless31.720.9Nucleotide Transformer
BacBench protein-protein interaction, linear probe AUROCBacBench protein-protein interaction dataset from STRING DBAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless63.956.5Nucleotide Transformer

What disagrees

Other studies
rank Nucleotide Transformer above Evo in 50 of 54 comparisons
This study
ranks Evo above Nucleotide Transformer in 3 of 4 comparisons
Compared
Nucleotide Transformer, Evo
Study that disagrees
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md
Other studies
14 studiesBacBench LICENSE file at commit 64b7d45, BacBench repository README at commit 64b7d45, Bacformer LICENSE file at commit 61dca95, Bacformer repository README at commit 61dca95, Hugging Face model card for macwiatrak/bacformer-causal-complete-genomes, Hugging Face model card for macwiatrak/bacformer-causal-MAG, Hugging Face model card for macwiatrak/bacformer-for-ppi-prediction, Hugging Face model card for macwiatrak/bacformer-large-masked-complete-genomes, Hugging Face model card for macwiatrak/bacformer-large-masked-MAG, Hugging Face model card for macwiatrak/bacformer-masked-complete-genomes, Hugging Face model card for macwiatrak/bacformer-masked-MAG, Bacformer preprint, bioRxiv 2025.07.20.665723 version 2, Bacformer preprint, bioRxiv 2025.07.20.665723 version 3, GENEB leaderboard (leaderboard.json, 13 task-type aggregates)
Rule
Ordering reversal; different-task
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (8)
StudyToolProtocol or benchmarkMetricValue
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdEvo 1.5 (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, AptamerSpearman (mean signed Spearman over the 2 Aptamer assays)0.0007
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdEvo 1.5 (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesSpearman (macro-mean over 3 ncRNA categories (equal category weights))0.138
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdEvo 1.5 (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, RibozymeSpearman (mean signed Spearman over the 26 Ribozyme assays)0.0278
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdEvo 1.5 (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, tRNASpearman (mean signed Spearman over the 3 tRNA assays)0.385
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdNucleotide Transformer (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, AptamerSpearman (mean signed Spearman over the 2 Aptamer assays)-0.0886
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdNucleotide Transformer (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, macro over 3 ncRNA categoriesSpearman (macro-mean over 3 ncRNA categories (equal category weights))0.12
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdNucleotide Transformer (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, RibozymeSpearman (mean signed Spearman over the 26 Ribozyme assays)0.133
RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.mdNucleotide Transformer (RNAGym fitness leaderboard v0.1.1)RNAGym v0.1.1 fitness, signed Spearman, tRNASpearman (mean signed Spearman over the 3 tRNA assays)0.317
In the other studies (108 results)
In the other studies
StudyToolProtocol or benchmarkMetricValue
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUPRCAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)44.06 ± 23.28
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUROCAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)68.22 ± 9.32
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression Pearson correlationPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)29.18 ± 17.26
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression R squaredR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)10.01 ± 10.06
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Nucleotide Transformer (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUPRCAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)59.70 ± 22.90
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Nucleotide Transformer (frozen embeddings with a trained linear head)BacBench antibiotic resistance, binary linear probe AUROCAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics)84.22 ± 6.47
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Nucleotide Transformer (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression Pearson correlationPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)51.19 ± 16.76
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Nucleotide Transformer (frozen embeddings with a trained linear head)BacBench antibiotic resistance, MIC regression R squaredR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage)27.74 ± 17.80
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUPRCAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)13.67 ± 22.40
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUROCAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)51.78 ± 4.16
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Nucleotide Transformer (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUPRCAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)25.96 ± 21.26
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Nucleotide Transformer (frozen embeddings with a trained linear head)BacBench essential gene prediction, linear probe AUROCAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes)65.68 ± 8.02
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUPRCAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)39.55 ± 33.36
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUROCAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)60.19 ± 11.04
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Nucleotide Transformer (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUPRCAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)43.05 ± 34.91
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Nucleotide Transformer (frozen embeddings with a trained linear head)BacBench phenotypic trait prediction, linear probe AUROCAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds)65.98 ± 11.54
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUPRCAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)20.93 ± 8.80
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Evo (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUROCAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)56.45 ± 3.14
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Nucleotide Transformer (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUPRCAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)31.66 ± 9.86
Bacformer preprint, bioRxiv 2025.07.20.665723 version 3Nucleotide Transformer (frozen embeddings with a trained linear head)BacBench protein-protein interaction, linear probe AUROCAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes)63.92 ± 2.98
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Chromatin Acc.')0.289
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Coding/NC')0.539
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'DNA Methyl.')0.0458
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Enhancers')0.288
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'lncRNA')0.0189
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; macro average: unweighted mean of the 13 category MCCs)0.194
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; micro average: mean over all 100 tasks)0.232
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Mouse Enh.')0.12
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Promoters')0.436
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Regulatory')0.125
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Species Clf.')0.207
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Splice Sites')0.0657
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'TF Binding')0.0386
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Virus/Phage')0.0941
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Chromatin Acc.')0.14
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Coding/NC')0.293
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'DNA Methyl.')0.0208
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Enhancers')0.165
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Histone Mod.')0.108
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'lncRNA')0.00964
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; macro average: unweighted mean of the 13 category MCCs)0.102
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; micro average: mean over all 100 tasks)0.126
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Mouse Enh.')0.0568
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Promoters')0.279
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Regulatory')0.0788
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Species Clf.')0.103
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Splice Sites')0.0643
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'TF Binding')0.0084
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Virus/Phage')-0.00343
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Chromatin Acc.')0.284
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Coding/NC')0.719
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'DNA Methyl.')0.0732
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Enhancers')0.37
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Histone Mod.')0.397
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'lncRNA')0.119
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; macro average: unweighted mean of the 13 category MCCs)0.298
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; micro average: mean over all 100 tasks)0.342
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Mouse Enh.')0.287
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Promoters')0.537
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Regulatory')0.269
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Species Clf.')0.285
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Splice Sites')0.16
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'TF Binding')0.173
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)evo-1-131k (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Virus/Phage')0.205
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Chromatin Acc.')0.41
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Coding/NC')0.667
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'DNA Methyl.')0.0575
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Enhancers')0.314
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'lncRNA')0.103
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; macro average: unweighted mean of the 13 category MCCs)0.293
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; micro average: mean over all 100 tasks)0.3
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Mouse Enh.')0.202
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Promoters')0.532
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Regulatory')0.152
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Species Clf.')0.512
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Splice Sites')0.242
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'TF Binding')0.201
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Virus/Phage')0.161
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Chromatin Acc.')0.169
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Coding/NC')0.251
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'DNA Methyl.')0.0179
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Enhancers')0.171
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Histone Mod.')0.0952
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'lncRNA')0.0454
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; macro average: unweighted mean of the 13 category MCCs)0.116
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; micro average: mean over all 100 tasks)0.136
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Mouse Enh.')0.0481
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Promoters')0.309
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Regulatory')0.0837
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Species Clf.')0.192
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Splice Sites')0.0768
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'TF Binding')0.0361
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Virus/Phage')0.0194
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Chromatin Acc.')0.554
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Coding/NC')0.864
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'DNA Methyl.')0.356
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Enhancers')0.487
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Histone Mod.')0.498
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'lncRNA')0.372
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; macro average: unweighted mean of the 13 category MCCs)0.573
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; micro average: mean over all 100 tasks)0.561
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Mouse Enh.')0.547
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Promoters')0.739
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Regulatory')0.488
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Species Clf.')0.752
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Splice Sites')0.652
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'TF Binding')0.573
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Virus/Phage')0.564

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-e05599ffec82fd2f · All contested results · How findings are made and followed up