rewirebio.iobenchmarks
Contested result

Nucleotide Transformer and HyenaDNA: TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) ranks HyenaDNA above Nucleotide Transformer in 2 of 2 comparisons; 4 of 4 other studies rank Nucleotide Transformer above HyenaDNA.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

5 papers rank Nucleotide Transformer above HyenaDNA in 77 of their 82 comparisons and HyenaDNA higher in 5. TraitGym official metrics ranks HyenaDNA above Nucleotide Transformer in all 2 of its comparisons.

Papers grouped by which system they rank higher: Nucleotide Transformer or HyenaDNARank Nucleotide Transformer higher: GENEB leaderboard, Nucleotide Transformer, A Phylogenetic Approach to Genomic…, Comparative Analysis of Deep Learning…. Rank HyenaDNA higher: TraitGym official metrics (disagrees). Mixed: Evaluating the representational power….Rank Nucleotide Transformer h…MixedRank HyenaDNA higherGENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)Nucleotide TransformerNucleotide Transformer: building and evaluating robust foundation models for human genomics (Nature Methods)A PhylogeneticApproach to Genomic…A Phylogenetic Approach to Genomic Language ModelingComparative Analysisof Deep Learning…Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsEvaluating therepresentational power…Evaluating the representational power of pre-trained DNA language models for regulatory genomicsTraitGym official metricsTraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)
  • Rank Nucleotide Transformer higher (4): GENEB leaderboard, Nucleotide Transformer, A Phylogenetic Approach to Genomic…, Comparative Analysis of Deep Learning…
  • Rank HyenaDNA higher (1): TraitGym official metrics (disagrees)
  • Mixed (1): Evaluating the representational power…

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Nucleotide Transformer or HyenaDNA higherTraitGym official metrics (disagrees): Nucleotide Transformer higher in 0, HyenaDNA higher in 2, 0 ties. GENEB leaderboard: Nucleotide Transformer higher in 43, HyenaDNA higher in 2, 0 ties. Nucleotide Transformer: Nucleotide Transformer higher in 19, HyenaDNA higher in 0, 0 ties. A Phylogenetic Approach to Genomic…: Nucleotide Transformer higher in 11, HyenaDNA higher in 1, 0 ties. Comparative Analysis of Deep Learning…: Nucleotide Transformer higher in 3, HyenaDNA higher in 1, 0 ties. Evaluating the representational power…: Nucleotide Transformer higher in 1, HyenaDNA higher in 1, 0 ties.Disagrees: TraitGym official metricsTraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)2 HyenaDNA higher2GENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)43 Nucleotide Transformer higher432 HyenaDNA higherNucleotide TransformerNucleotide Transformer: building and evaluating robust foundation models for human genomics (Nature Methods)19 Nucleotide Transformer higher19A Phylogenetic Approach to Genomic…A Phylogenetic Approach to Genomic Language Modeling11 Nucleotide Transformer higher111 HyenaDNA higherComparative Analysis of Deep Learning…Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants3 Nucleotide Transformer higher31 HyenaDNA higher1Evaluating the representational power…Evaluating the representational power of pre-trained DNA language models for regulatory genomics1 Nucleotide Transformer higher11 HyenaDNA higher1
Nucleotide Transformer higherTieHyenaDNA higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in TraitGym official metrics (2)
ProtocolDatasetMetricNucleotide TransformerHyenaDNAHigher
TraitGym Complex traits, zero-shot, AUPRC by chromosome weighted averageTraitGym Complex traits, matched 9 controlsAUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all'), unitless0.10.111HyenaDNA
TraitGym Mendelian traits, zero-shot, AUPRC by chromosome weighted averageTraitGym Mendelian traits, matched 9 controlsAUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all'), unitless0.09810.106HyenaDNA
Comparisons in GENEB leaderboard (45)
ProtocolDatasetMetricNucleotide TransformerHyenaDNAHigher
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Chromatin Acc.'), unitless0.410.34Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Coding/NC'), unitless0.6670.554Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'DNA Methyl.'), unitless0.05750.0554Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Enhancers'), unitless0.3140.264Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Histone Mod.'), unitless0.2580.198Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Mouse Enh.'), unitless0.2020.0433Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Promoters'), unitless0.5320.462Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Regulatory'), unitless0.1520.142Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Species Clf.'), unitless0.5120.385Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Splice Sites'), unitless0.2420.125Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'TF Binding'), unitless0.2010.0921Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Virus/Phage'), unitless0.1610.101Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'lncRNA'), unitless0.1030.0278Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; macro average: unweighted mean of the 13 category MCCs), unitless0.2930.215Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; micro average: mean over all 100 tasks), unitless0.30.229Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Chromatin Acc.'), unitless0.1690.129Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Coding/NC'), unitless0.2510.311HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'DNA Methyl.'), unitless0.01790.00611Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Enhancers'), unitless0.1710.133Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Histone Mod.'), unitless0.09520.0859Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Mouse Enh.'), unitless0.04810.0165Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Promoters'), unitless0.3090.272Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Regulatory'), unitless0.08370.0598Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Species Clf.'), unitless0.1920.18Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Splice Sites'), unitless0.07680.0644Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'TF Binding'), unitless0.03610.0298Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Virus/Phage'), unitless0.01940.0421HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'lncRNA'), unitless0.04540.0213Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; macro average: unweighted mean of the 13 category MCCs), unitless0.1160.104Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; micro average: mean over all 100 tasks), unitless0.1360.117Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Chromatin Acc.'), unitless0.5540.54Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Coding/NC'), unitless0.8640.712Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'DNA Methyl.'), unitless0.3560.169Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Enhancers'), unitless0.4870.436Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Histone Mod.'), unitless0.4980.448Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Mouse Enh.'), unitless0.5470.39Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Promoters'), unitless0.7390.696Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Regulatory'), unitless0.4880.423Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Species Clf.'), unitless0.7520.666Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Splice Sites'), unitless0.6520.401Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'TF Binding'), unitless0.5730.532Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Virus/Phage'), unitless0.5640.346Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'lncRNA'), unitless0.3720.158Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; macro average: unweighted mean of the 13 category MCCs), unitless0.5730.455Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; micro average: mean over all 100 tasks), unitless0.5610.468Nucleotide Transformer
Comparisons in Nucleotide Transformer (19)
ProtocolDatasetMetricNucleotide TransformerHyenaDNAHigher
NT benchmark Enhancer, fine-tuning (PEFT)NT benchmark EnhancerMCC (median over 10 cross-validation folds), unitless0.5040.475Nucleotide Transformer
NT benchmark Enhancer types, fine-tuning (PEFT)NT benchmark Enhancer typesMCC (median over 10 cross-validation folds), unitless0.4690.441Nucleotide Transformer
NT benchmark H2AFZ, fine-tuning (PEFT)NT benchmark H2AFZMCC (median over 10 cross-validation folds), unitless0.4780.455Nucleotide Transformer
NT benchmark H3K27ac, fine-tuning (PEFT)NT benchmark H3K27acMCC (median over 10 cross-validation folds), unitless0.4860.423Nucleotide Transformer
NT benchmark H3K27me3, fine-tuning (PEFT)NT benchmark H3K27me3MCC (median over 10 cross-validation folds), unitless0.6030.541Nucleotide Transformer
NT benchmark H3K36me3, fine-tuning (PEFT)NT benchmark H3K36me3MCC (median over 10 cross-validation folds), unitless0.6320.543Nucleotide Transformer
NT benchmark H3K4me1, fine-tuning (PEFT)NT benchmark H3K4me1MCC (median over 10 cross-validation folds), unitless0.4910.43Nucleotide Transformer
NT benchmark H3K4me2, fine-tuning (PEFT)NT benchmark H3K4me2MCC (median over 10 cross-validation folds), unitless0.5690.521Nucleotide Transformer
NT benchmark H3K4me3, fine-tuning (PEFT)NT benchmark H3K4me3MCC (median over 10 cross-validation folds), unitless0.6150.596Nucleotide Transformer
NT benchmark H3K9ac, fine-tuning (PEFT)NT benchmark H3K9acMCC (median over 10 cross-validation folds), unitless0.5290.484Nucleotide Transformer
NT benchmark H3K9me3, fine-tuning (PEFT)NT benchmark H3K9me3MCC (median over 10 cross-validation folds), unitless0.4830.375Nucleotide Transformer
NT benchmark H4K20me1, fine-tuning (PEFT)NT benchmark H4K20me1MCC (median over 10 cross-validation folds), unitless0.6590.58Nucleotide Transformer
NT benchmark normalised summed performance (Supplementary Table 9)NT benchmark, all 18 tasksMCC (mean over three task categories of median cross-validation MCC), unitless0.7380.668Nucleotide Transformer
NT benchmark Promoter all, fine-tuning (PEFT)NT benchmark Promoter allMCC (median over 10 cross-validation folds), unitless0.7080.693Nucleotide Transformer
NT benchmark Promoter non-TATA, fine-tuning (PEFT)NT benchmark Promoter non-TATAMCC (median over 10 cross-validation folds), unitless0.7580.723Nucleotide Transformer
NT benchmark Promoter TATA, fine-tuning (PEFT)NT benchmark Promoter TATAMCC (median over 10 cross-validation folds), unitless0.8020.648Nucleotide Transformer
NT benchmark Splice acceptor, fine-tuning (PEFT)NT benchmark Splice acceptorMCC (median over 10 cross-validation folds), unitless0.9520.815Nucleotide Transformer
NT benchmark Splice donor, fine-tuning (PEFT)NT benchmark Splice donorMCC (median over 10 cross-validation folds), unitless0.9630.943Nucleotide Transformer
NT benchmark Splice site all, fine-tuning (PEFT)NT benchmark Splice site allMCC (median over 10 cross-validation folds), unitless0.9560.854Nucleotide Transformer
Comparisons in A Phylogenetic Approach to Genomic… (12)
ProtocolDatasetMetricNucleotide TransformerHyenaDNAHigher
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Downstream of Transcript variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Downstream of Transcript), unitless0.580.55Nucleotide Transformer
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Intronic variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Intronic), unitless0.610.48Nucleotide Transformer
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Missense variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Missense), unitless0.640.54Nucleotide Transformer
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Noncoding variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Noncoding), unitless0.540.47Nucleotide Transformer
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Splice Acceptor variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Acceptor), unitless0.620.51Nucleotide Transformer
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Splice Donor variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Donor), unitless0.670.52Nucleotide Transformer
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Start Codon variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Start Codon), unitless0.690.43Nucleotide Transformer
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Stop Lost variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Stop Lost), unitless0.550.48Nucleotide Transformer
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Synonymous variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Synonymous), unitless0.550.43Nucleotide Transformer
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Upstream of Transcript variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Upstream of Transcript), unitless0.310.51HyenaDNA
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar 3’ UTR variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 3’ UTR), unitless0.660.48Nucleotide Transformer
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar 5’ UTR variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 5’ UTR), unitless0.550.48Nucleotide Transformer
Comparisons in Comparative Analysis of Deep Learning… (4)
ProtocolDatasetMetricNucleotide TransformerHyenaDNAHigher
Allelic reporter effect correlation in HeLa (Manzo et al. 2025 Table 1)HeLa regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.0870.055Nucleotide Transformer
Allelic reporter effect correlation in HepG2 (Manzo et al. 2025 Table 1)HepG2 regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.1120.134HyenaDNA
Allelic reporter effect correlation in K562 (Manzo et al. 2025 Table 1)K562 regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.1470.077Nucleotide Transformer
Allelic reporter effect correlation in NPC (Manzo et al. 2025 Table 1)NPC regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.005-0.007Nucleotide Transformer
Comparisons in Evaluating the representational power… (2)
ProtocolDatasetMetricNucleotide TransformerHyenaDNAHigher
CAGI5 saturation MPRA variant effect correlation, HepG2 (Tang et al. 2025 Table 1)CAGI5 saturation mutagenesis MPRA, four CREs (as used by Tang et al. 2025)Pearson (predicted versus MPRA-measured saturation mutagenesis variant effect; mean over three CREs), unitless0.1250.064Nucleotide Transformer
CAGI5 saturation MPRA variant effect correlation, K562 (Tang et al. 2025 Table 1)CAGI5 saturation mutagenesis MPRA, four CREs (as used by Tang et al. 2025)Pearson (predicted versus MPRA-measured saturation mutagenesis variant effect; one CRE), unitless0.0070.021HyenaDNA

What disagrees

Other studies
rank Nucleotide Transformer above HyenaDNA in 76 of 80 comparisons
This study
ranks HyenaDNA above Nucleotide Transformer in 2 of 2 comparisons
Compared
Nucleotide Transformer, HyenaDNA
Study that disagrees
TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym), Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)
Other studies
7 studiesNucleotide Transformer: building and evaluating robust foundation models for human genomics (Nature Methods), GENEB leaderboard (leaderboard.json, 13 task-type aggregates), Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12), A Phylogenetic Approach to Genomic Language Modeling, Springer chapter page for the RECOMB 2025 version of record, Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1), Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants
Rule
Ordering reversal; unknown
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the other studies (160 results)
In the other studies
StudyToolProtocol or benchmarkMetricValue
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Chromatin Acc.')0.34
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Coding/NC')0.554
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'DNA Methyl.')0.0554
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Enhancers')0.264
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Histone Mod.')0.198
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'lncRNA')0.0278
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; macro average: unweighted mean of the 13 category MCCs)0.215
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; micro average: mean over all 100 tasks)0.229
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Mouse Enh.')0.0433
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Promoters')0.462
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Regulatory')0.142
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Species Clf.')0.385
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Splice Sites')0.125
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'TF Binding')0.0921
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Virus/Phage')0.101
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Chromatin Acc.')0.129
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Coding/NC')0.311
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'DNA Methyl.')0.00611
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Enhancers')0.133
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Histone Mod.')0.0859
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'lncRNA')0.0213
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; macro average: unweighted mean of the 13 category MCCs)0.104
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; micro average: mean over all 100 tasks)0.117
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Mouse Enh.')0.0165
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Promoters')0.272
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Regulatory')0.0598
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Species Clf.')0.18
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Splice Sites')0.0644
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'TF Binding')0.0298
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Virus/Phage')0.0421
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Chromatin Acc.')0.54
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Coding/NC')0.712
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'DNA Methyl.')0.169
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Enhancers')0.436
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Histone Mod.')0.448
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'lncRNA')0.158
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; macro average: unweighted mean of the 13 category MCCs)0.455
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; micro average: mean over all 100 tasks)0.468
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Mouse Enh.')0.39
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Promoters')0.696
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Regulatory')0.423
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Species Clf.')0.666
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Splice Sites')0.401
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'TF Binding')0.532
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Virus/Phage')0.346
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Chromatin Acc.')0.41
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Coding/NC')0.667
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'DNA Methyl.')0.0575
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Enhancers')0.314
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Histone Mod.')0.258
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'lncRNA')0.103
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; macro average: unweighted mean of the 13 category MCCs)0.293
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; micro average: mean over all 100 tasks)0.3
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Mouse Enh.')0.202
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Promoters')0.532
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Regulatory')0.152
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Species Clf.')0.512
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Splice Sites')0.242
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'TF Binding')0.201
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Virus/Phage')0.161
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Chromatin Acc.')0.169
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Coding/NC')0.251
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'DNA Methyl.')0.0179
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Enhancers')0.171
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Histone Mod.')0.0952
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'lncRNA')0.0454
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; macro average: unweighted mean of the 13 category MCCs)0.116
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; micro average: mean over all 100 tasks)0.136
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Mouse Enh.')0.0481
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Promoters')0.309
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Regulatory')0.0837
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Species Clf.')0.192
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Splice Sites')0.0768
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'TF Binding')0.0361
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Virus/Phage')0.0194
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Chromatin Acc.')0.554
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Coding/NC')0.864
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'DNA Methyl.')0.356
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Enhancers')0.487
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Histone Mod.')0.498
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'lncRNA')0.372
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; macro average: unweighted mean of the 13 category MCCs)0.573
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; micro average: mean over all 100 tasks)0.561
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Mouse Enh.')0.547
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Promoters')0.739
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Regulatory')0.488
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Species Clf.')0.752
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Splice Sites')0.652
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'TF Binding')0.573
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)NT-2.5B-MS (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Virus/Phage')0.564
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark Enhancer, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.475 (± 0.006)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark Enhancer types, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.441 (± 0.01)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark H2AFZ, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.455 (± 0.015)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K27ac, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.423 (± 0.017)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K27me3, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.541 (± 0.018)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K36me3, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.543 (± 0.01)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K4me1, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.43 (± 0.014)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K4me2, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.521 (± 0.024)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K4me3, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.596 (± 0.015)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K9ac, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.484 (± 0.022)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K9me3, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.375 (± 0.026)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark H4K20me1, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.58 (± 0.009)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark Promoter all, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.693 (± 0.016)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark Promoter non-TATA, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.723 (± 0.013)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark Promoter TATA, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.648 (± 0.044)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark Splice acceptor, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.815 (± 0.049)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark Splice donor, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.943 (± 0.024)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark Splice site all, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.854 (± 0.053)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)HyenaDNA-1KB, parameter-efficient fine-tuning (NT benchmark)NT benchmark normalised summed performance (Supplementary Table 9)MCC (mean over three task categories of median cross-validation MCC)0.668
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark Enhancer, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.504 (± 0.009)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark Enhancer types, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.469 (± 0.005)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark H2AFZ, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.478 (± 0.012)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K27ac, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.486 (± 0.023)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K27me3, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.603 (± 0.009)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K36me3, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.632 (± 0.008)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K4me1, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.491 (± 0.015)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K4me2, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.569 (± 0.014)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K4me3, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.615 (± 0.017)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K9ac, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.529 (± 0.012)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark H3K9me3, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.483 (± 0.013)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark H4K20me1, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.659 (± 0.008)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark Promoter all, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.708 (± 0.008)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark Promoter non-TATA, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.758 (± 0.007)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark Promoter TATA, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.802 (± 0.03)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark Splice acceptor, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.952 (± 0.004)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark Splice donor, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.963 (± 0.001)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark Splice site all, fine-tuning (PEFT)MCC (median over 10 cross-validation folds)0.956 (± 0.004)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)NT benchmark normalised summed performance (Supplementary Table 9)MCC (mean over three task categories of median cross-validation MCC)0.738
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Downstream of Transcript)0.55
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Downstream of Transcript)0.58
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Intronic)0.48
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Intronic)0.61
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Missense)0.54
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Missense)0.64
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Noncoding)0.47
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Noncoding)0.54
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Acceptor)0.51
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Acceptor)0.62
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Donor)0.52
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Donor)0.67
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Start Codon)0.43
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Start Codon)0.69
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Stop Lost)0.48
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Stop Lost)0.55
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Synonymous)0.43
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Synonymous)0.55
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Upstream of Transcript)0.51
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Upstream of Transcript)0.31
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 3’ UTR)0.48
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 3’ UTR)0.66
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 5’ UTR)0.48
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Nucleotide Transformer v2 500M multi-species, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 5’ UTR)0.55
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsHyenadna 450 k (Manzo et al. 2025)Allelic reporter effect correlation in HeLa (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.055
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsNT 2.5b-1000g (Manzo et al. 2025)Allelic reporter effect correlation in HeLa (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.087
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsHyenadna 450 k (Manzo et al. 2025)Allelic reporter effect correlation in HepG2 (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.134
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsNT 2.5b-1000g (Manzo et al. 2025)Allelic reporter effect correlation in HepG2 (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.112
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsHyenadna 450 k (Manzo et al. 2025)Allelic reporter effect correlation in K562 (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.077
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsNT 2.5b-1000g (Manzo et al. 2025)Allelic reporter effect correlation in K562 (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.147
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsHyenadna 450 k (Manzo et al. 2025)Allelic reporter effect correlation in NPC (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)−0.007
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsNT 2.5b-1000g (Manzo et al. 2025)Allelic reporter effect correlation in NPC (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.005

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-dec6ace9a498c06c · All contested results · How findings are made and followed up