Nucleotide Transformer and HyenaDNA: TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) reverses the order
- Low severity
- Not yet reviewed
- Ordering reversal
- unknown
TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) ranks HyenaDNA above Nucleotide Transformer in 2 of 2 comparisons; 4 of 4 other studies rank Nucleotide Transformer above HyenaDNA.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
5 papers rank Nucleotide Transformer above HyenaDNA in 77 of their 82 comparisons and HyenaDNA higher in 5. TraitGym official metrics ranks HyenaDNA above Nucleotide Transformer in all 2 of its comparisons.
- Rank Nucleotide Transformer higher (4): GENEB leaderboard, Nucleotide Transformer, A Phylogenetic Approach to Genomic…, Comparative Analysis of Deep Learning…
- Rank HyenaDNA higher (1): TraitGym official metrics (disagrees)
- Mixed (1): Evaluating the representational power…
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks Nucleotide Transformer higher | Ranks HyenaDNA higher | Ties |
|---|---|---|---|---|
| TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) and 1 more record · DOI | Disagrees | 0 | 2 | 0 |
| GENEB leaderboard (leaderboard.json, 13 task-type aggregates) · Repository | Agrees | 43 | 2 | 0 |
| Nucleotide Transformer: building and evaluating robust foundation models for human genomics (Nature Methods) and 1 more record · DOI | Agrees | 19 | 0 | 0 |
| A Phylogenetic Approach to Genomic Language Modeling and 2 more records · DOI | Agrees | 11 | 1 | 0 |
| Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants · DOI | Agrees | 3 | 1 | 0 |
| Evaluating the representational power of pre-trained DNA language models for regulatory genomics · DOI | Agrees | 1 | 1 | 0 |
Comparisons in TraitGym official metrics (2)
| Protocol | Dataset | Metric | Nucleotide Transformer | HyenaDNA | Higher |
|---|---|---|---|---|---|
| TraitGym Complex traits, zero-shot, AUPRC by chromosome weighted average | TraitGym Complex traits, matched 9 controls | AUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all'), unitless | 0.1 | 0.111 | HyenaDNA |
| TraitGym Mendelian traits, zero-shot, AUPRC by chromosome weighted average | TraitGym Mendelian traits, matched 9 controls | AUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all'), unitless | 0.0981 | 0.106 | HyenaDNA |
Comparisons in GENEB leaderboard (45)
| Protocol | Dataset | Metric | Nucleotide Transformer | HyenaDNA | Higher |
|---|---|---|---|---|---|
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Chromatin Acc.'), unitless | 0.41 | 0.34 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Coding/NC'), unitless | 0.667 | 0.554 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'DNA Methyl.'), unitless | 0.0575 | 0.0554 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Enhancers'), unitless | 0.314 | 0.264 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Histone Mod.'), unitless | 0.258 | 0.198 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Mouse Enh.'), unitless | 0.202 | 0.0433 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Promoters'), unitless | 0.532 | 0.462 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Regulatory'), unitless | 0.152 | 0.142 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Species Clf.'), unitless | 0.512 | 0.385 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Splice Sites'), unitless | 0.242 | 0.125 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'TF Binding'), unitless | 0.201 | 0.0921 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'Virus/Phage'), unitless | 0.161 | 0.101 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; category 'lncRNA'), unitless | 0.103 | 0.0278 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; macro average: unweighted mean of the 13 category MCCs), unitless | 0.293 | 0.215 | Nucleotide Transformer |
| GENEB leaderboard, 10-shot, MCC aggregates | GENEB 100 tasks | MCC (10-shot; micro average: mean over all 100 tasks), unitless | 0.3 | 0.229 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Chromatin Acc.'), unitless | 0.169 | 0.129 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Coding/NC'), unitless | 0.251 | 0.311 | HyenaDNA |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'DNA Methyl.'), unitless | 0.0179 | 0.00611 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Enhancers'), unitless | 0.171 | 0.133 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Histone Mod.'), unitless | 0.0952 | 0.0859 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Mouse Enh.'), unitless | 0.0481 | 0.0165 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Promoters'), unitless | 0.309 | 0.272 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Regulatory'), unitless | 0.0837 | 0.0598 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Species Clf.'), unitless | 0.192 | 0.18 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Splice Sites'), unitless | 0.0768 | 0.0644 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'TF Binding'), unitless | 0.0361 | 0.0298 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'Virus/Phage'), unitless | 0.0194 | 0.0421 | HyenaDNA |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; category 'lncRNA'), unitless | 0.0454 | 0.0213 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; macro average: unweighted mean of the 13 category MCCs), unitless | 0.116 | 0.104 | Nucleotide Transformer |
| GENEB leaderboard, 1-shot, MCC aggregates | GENEB 100 tasks | MCC (1-shot; micro average: mean over all 100 tasks), unitless | 0.136 | 0.117 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Chromatin Acc.'), unitless | 0.554 | 0.54 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Coding/NC'), unitless | 0.864 | 0.712 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'DNA Methyl.'), unitless | 0.356 | 0.169 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Enhancers'), unitless | 0.487 | 0.436 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Histone Mod.'), unitless | 0.498 | 0.448 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Mouse Enh.'), unitless | 0.547 | 0.39 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Promoters'), unitless | 0.739 | 0.696 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Regulatory'), unitless | 0.488 | 0.423 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Species Clf.'), unitless | 0.752 | 0.666 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Splice Sites'), unitless | 0.652 | 0.401 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'TF Binding'), unitless | 0.573 | 0.532 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'Virus/Phage'), unitless | 0.564 | 0.346 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; category 'lncRNA'), unitless | 0.372 | 0.158 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; macro average: unweighted mean of the 13 category MCCs), unitless | 0.573 | 0.455 | Nucleotide Transformer |
| GENEB leaderboard, full training data, MCC aggregates | GENEB 100 tasks | MCC (full training data; micro average: mean over all 100 tasks), unitless | 0.561 | 0.468 | Nucleotide Transformer |
Comparisons in Nucleotide Transformer (19)
| Protocol | Dataset | Metric | Nucleotide Transformer | HyenaDNA | Higher |
|---|---|---|---|---|---|
| NT benchmark Enhancer, fine-tuning (PEFT) | NT benchmark Enhancer | MCC (median over 10 cross-validation folds), unitless | 0.504 | 0.475 | Nucleotide Transformer |
| NT benchmark Enhancer types, fine-tuning (PEFT) | NT benchmark Enhancer types | MCC (median over 10 cross-validation folds), unitless | 0.469 | 0.441 | Nucleotide Transformer |
| NT benchmark H2AFZ, fine-tuning (PEFT) | NT benchmark H2AFZ | MCC (median over 10 cross-validation folds), unitless | 0.478 | 0.455 | Nucleotide Transformer |
| NT benchmark H3K27ac, fine-tuning (PEFT) | NT benchmark H3K27ac | MCC (median over 10 cross-validation folds), unitless | 0.486 | 0.423 | Nucleotide Transformer |
| NT benchmark H3K27me3, fine-tuning (PEFT) | NT benchmark H3K27me3 | MCC (median over 10 cross-validation folds), unitless | 0.603 | 0.541 | Nucleotide Transformer |
| NT benchmark H3K36me3, fine-tuning (PEFT) | NT benchmark H3K36me3 | MCC (median over 10 cross-validation folds), unitless | 0.632 | 0.543 | Nucleotide Transformer |
| NT benchmark H3K4me1, fine-tuning (PEFT) | NT benchmark H3K4me1 | MCC (median over 10 cross-validation folds), unitless | 0.491 | 0.43 | Nucleotide Transformer |
| NT benchmark H3K4me2, fine-tuning (PEFT) | NT benchmark H3K4me2 | MCC (median over 10 cross-validation folds), unitless | 0.569 | 0.521 | Nucleotide Transformer |
| NT benchmark H3K4me3, fine-tuning (PEFT) | NT benchmark H3K4me3 | MCC (median over 10 cross-validation folds), unitless | 0.615 | 0.596 | Nucleotide Transformer |
| NT benchmark H3K9ac, fine-tuning (PEFT) | NT benchmark H3K9ac | MCC (median over 10 cross-validation folds), unitless | 0.529 | 0.484 | Nucleotide Transformer |
| NT benchmark H3K9me3, fine-tuning (PEFT) | NT benchmark H3K9me3 | MCC (median over 10 cross-validation folds), unitless | 0.483 | 0.375 | Nucleotide Transformer |
| NT benchmark H4K20me1, fine-tuning (PEFT) | NT benchmark H4K20me1 | MCC (median over 10 cross-validation folds), unitless | 0.659 | 0.58 | Nucleotide Transformer |
| NT benchmark normalised summed performance (Supplementary Table 9) | NT benchmark, all 18 tasks | MCC (mean over three task categories of median cross-validation MCC), unitless | 0.738 | 0.668 | Nucleotide Transformer |
| NT benchmark Promoter all, fine-tuning (PEFT) | NT benchmark Promoter all | MCC (median over 10 cross-validation folds), unitless | 0.708 | 0.693 | Nucleotide Transformer |
| NT benchmark Promoter non-TATA, fine-tuning (PEFT) | NT benchmark Promoter non-TATA | MCC (median over 10 cross-validation folds), unitless | 0.758 | 0.723 | Nucleotide Transformer |
| NT benchmark Promoter TATA, fine-tuning (PEFT) | NT benchmark Promoter TATA | MCC (median over 10 cross-validation folds), unitless | 0.802 | 0.648 | Nucleotide Transformer |
| NT benchmark Splice acceptor, fine-tuning (PEFT) | NT benchmark Splice acceptor | MCC (median over 10 cross-validation folds), unitless | 0.952 | 0.815 | Nucleotide Transformer |
| NT benchmark Splice donor, fine-tuning (PEFT) | NT benchmark Splice donor | MCC (median over 10 cross-validation folds), unitless | 0.963 | 0.943 | Nucleotide Transformer |
| NT benchmark Splice site all, fine-tuning (PEFT) | NT benchmark Splice site all | MCC (median over 10 cross-validation folds), unitless | 0.956 | 0.854 | Nucleotide Transformer |
Comparisons in A Phylogenetic Approach to Genomic… (12)
| Protocol | Dataset | Metric | Nucleotide Transformer | HyenaDNA | Higher |
|---|---|---|---|---|---|
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Downstream of Transcript variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Downstream of Transcript), unitless | 0.58 | 0.55 | Nucleotide Transformer |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Intronic variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Intronic), unitless | 0.61 | 0.48 | Nucleotide Transformer |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Missense variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Missense), unitless | 0.64 | 0.54 | Nucleotide Transformer |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Noncoding variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Noncoding), unitless | 0.54 | 0.47 | Nucleotide Transformer |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Splice Acceptor variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Acceptor), unitless | 0.62 | 0.51 | Nucleotide Transformer |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Splice Donor variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Donor), unitless | 0.67 | 0.52 | Nucleotide Transformer |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Start Codon variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Start Codon), unitless | 0.69 | 0.43 | Nucleotide Transformer |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Stop Lost variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Stop Lost), unitless | 0.55 | 0.48 | Nucleotide Transformer |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Synonymous variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Synonymous), unitless | 0.55 | 0.43 | Nucleotide Transformer |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Upstream of Transcript variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Upstream of Transcript), unitless | 0.31 | 0.51 | HyenaDNA |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar 3’ UTR variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 3’ UTR), unitless | 0.66 | 0.48 | Nucleotide Transformer |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar 5’ UTR variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 5’ UTR), unitless | 0.55 | 0.48 | Nucleotide Transformer |
Comparisons in Comparative Analysis of Deep Learning… (4)
Comparisons in Evaluating the representational power… (2)
| Protocol | Dataset | Metric | Nucleotide Transformer | HyenaDNA | Higher |
|---|---|---|---|---|---|
| CAGI5 saturation MPRA variant effect correlation, HepG2 (Tang et al. 2025 Table 1) | CAGI5 saturation mutagenesis MPRA, four CREs (as used by Tang et al. 2025) | Pearson (predicted versus MPRA-measured saturation mutagenesis variant effect; mean over three CREs), unitless | 0.125 | 0.064 | Nucleotide Transformer |
| CAGI5 saturation MPRA variant effect correlation, K562 (Tang et al. 2025 Table 1) | CAGI5 saturation mutagenesis MPRA, four CREs (as used by Tang et al. 2025) | Pearson (predicted versus MPRA-measured saturation mutagenesis variant effect; one CRE), unitless | 0.007 | 0.021 | HyenaDNA |
What disagrees
- Other studies
- rank Nucleotide Transformer above HyenaDNA in 76 of 80 comparisons
- This study
- ranks HyenaDNA above Nucleotide Transformer in 2 of 2 comparisons
- Compared
- Nucleotide Transformer, HyenaDNA
- Study that disagrees
- TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym), Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)
- Other studies
7 studies
Nucleotide Transformer: building and evaluating robust foundation models for human genomics (Nature Methods), GENEB leaderboard (leaderboard.json, 13 task-type aggregates), Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12), A Phylogenetic Approach to Genomic Language Modeling, Springer chapter page for the RECOMB 2025 version of record, Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1), Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants- Rule
- Ordering reversal; unknown
- Model identity
- Matched across studies partly through links not yet reviewed
The compared numbers
Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.
In the other studies (160 results)
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-dec6ace9a498c06c · All contested results · How findings are made and followed up