rewirebio.iobenchmarks
Contested result

Caduceus and HyenaDNA: A Phylogenetic Approach to Genomic Language Modeling reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

A Phylogenetic Approach to Genomic Language Modeling ranks Caduceus above HyenaDNA in 9 of 12 comparisons; 2 of 2 other studies rank HyenaDNA above Caduceus.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

5 papers rank HyenaDNA above Caduceus in 63 of their 133 comparisons and Caduceus higher in 49; 21 are ties. Of those papers, DNALONGBENCH ranks Caduceus higher on its own (16 of 27). A Phylogenetic Approach to Genomic… ranks Caduceus above HyenaDNA in 9 of its 12 comparisons and HyenaDNA higher in 3.

Papers grouped by which system they rank higher: HyenaDNA or CaduceusRank HyenaDNA higher: GENEB leaderboard, TraitGym official metrics. Rank Caduceus higher: A Phylogenetic Approach to Genomic… (disagrees), DNALONGBENCH. Mixed: GenBench, Comparative Analysis of Deep Learning….Rank HyenaDNA higherMixedRank Caduceus higherGENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)TraitGym official metricsTraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)GenBenchGenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsComparative Analysisof Deep Learning…Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsA PhylogeneticApproach to Genomic…A Phylogenetic Approach to Genomic Language ModelingDNALONGBENCHDNALONGBENCH: a benchmark suite for long-range DNA prediction tasks
  • Rank HyenaDNA higher (2): GENEB leaderboard, TraitGym official metrics
  • Rank Caduceus higher (2): A Phylogenetic Approach to Genomic… (disagrees), DNALONGBENCH
  • Mixed (2): GenBench, Comparative Analysis of Deep Learning…

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores HyenaDNA or Caduceus higherA Phylogenetic Approach to Genomic… (disagrees): HyenaDNA higher in 3, Caduceus higher in 9, 0 ties. GenBench: HyenaDNA higher in 16, Caduceus higher in 23, 16 ties. GENEB leaderboard: HyenaDNA higher in 37, Caduceus higher in 8, 0 ties. DNALONGBENCH: HyenaDNA higher in 6, Caduceus higher in 16, 5 ties. Comparative Analysis of Deep Learning…: HyenaDNA higher in 2, Caduceus higher in 2, 0 ties. TraitGym official metrics: HyenaDNA higher in 2, Caduceus higher in 0, 0 ties.Disagrees: A Phylogenetic Approach to Genomic…A Phylogenetic Approach to Genomic Language Modeling3 HyenaDNA higher39 Caduceus higher9GenBenchGenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models16 HyenaDNA higher1616 tie1623 Caduceus higher23GENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)37 HyenaDNA higher378 Caduceus higher8DNALONGBENCHDNALONGBENCH: a benchmark suite for long-range DNA prediction tasks6 HyenaDNA higher65 tie516 Caduceus higher16Comparative Analysis of Deep Learning…Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants2 HyenaDNA higher22 Caduceus higher2TraitGym official metricsTraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)2 HyenaDNA higher2
HyenaDNA higherTieCaduceus higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in A Phylogenetic Approach to Genomic… (12)
ProtocolDatasetMetricHyenaDNACaduceusHigher
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Downstream of Transcript variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Downstream of Transcript), unitless0.550.72Caduceus
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Intronic variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Intronic), unitless0.480.64Caduceus
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Missense variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Missense), unitless0.540.55Caduceus
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Noncoding variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Noncoding), unitless0.470.59Caduceus
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Splice Acceptor variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Acceptor), unitless0.510.61Caduceus
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Splice Donor variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Donor), unitless0.520.68Caduceus
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Start Codon variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Start Codon), unitless0.430.41HyenaDNA
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Stop Lost variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Stop Lost), unitless0.480.43HyenaDNA
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Synonymous variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Synonymous), unitless0.430.55Caduceus
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Upstream of Transcript variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Upstream of Transcript), unitless0.510.4HyenaDNA
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar 3’ UTR variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 3’ UTR), unitless0.480.63Caduceus
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar 5’ UTR variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 5’ UTR), unitless0.480.61Caduceus
Comparisons in GenBench (55)
ProtocolDatasetMetricHyenaDNACaduceusHigher
GenBench Bulk RNA expressionGenBench Bulk RNA expressionMSE, unitless0.4530.51HyenaDNA
GenBench Bulk RNA expressionGenBench Bulk RNA expressionSpearman, unitless0.7550.746HyenaDNA
GenBench Drosophila enhancer activity (dev)GenBench Drosophila enhancer activity (dev)Pearson, unitless0.470.443HyenaDNA
GenBench Drosophila enhancer activity (hk)GenBench Drosophila enhancer activity (hk)Pearson, unitless0.5520.53HyenaDNA
GenBench Drosophila enhancer activity (Mean)GenBench Drosophila enhancer activity (Mean)Pearson (mean of dev and hk as printed), unitless0.5110.486HyenaDNA
GenBench Genomic structure prediction (H1-ESC)GenBench Genomic structure prediction (H1-ESC)MSE, unitless0.01840.0168Caduceus
GenBench Genomic structure prediction (H1-ESC)GenBench Genomic structure prediction (H1-ESC)Pearson, unitless0.4360.502Caduceus
GenBench Genomic structure prediction (HFF)GenBench Genomic structure prediction (HFF)MSE, unitless0.1010.105HyenaDNA
GenBench Genomic structure prediction (HFF)GenBench Genomic structure prediction (HFF)Pearson, unitless0.310.354Caduceus
GenBench Long-range Promoters PredictionGenBench Long-range Promoters PredictionAccuracy, fraction0.8870.93Caduceus
GenBench Long-range Species ClassificationGenBench Long-range Species ClassificationAccuracy, fraction0.6770.711Caduceus
GenBench Long-range splice site prediction (Mean)GenBench Long-range splice site prediction (Mean)AUROC (mean of donor and acceptor as printed), unitless0.6480.691Caduceus
GenBench Long-range splice site prediction (Splice acceptor)GenBench Long-range splice site prediction (Splice acceptor)AUROC, unitless0.7230.74Caduceus
GenBench Long-range splice site prediction (Splice donar)GenBench Long-range splice site prediction (Splice donar)AUROC, unitless0.5740.642Caduceus
GenBench Coding vs IntergenomicGenBench Coding vs IntergenomicAccuracy (task-level value as printed in Table 4), fraction0.910.937Caduceus
GenBench Human Core Promoter DetectionGenBench Human Core Promoter DetectionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows), fraction0.8440.851Tie
GenBench Human Core Promoter Detection, dataset allGenBench Human Core Promoter Detection, dataset allAccuracy, fraction0.8310.834Tie
GenBench Human Core Promoter Detection, dataset notataGenBench Human Core Promoter Detection, dataset notataAccuracy, fraction0.8350.843Tie
GenBench Human Core Promoter Detection, dataset tataGenBench Human Core Promoter Detection, dataset tataAccuracy, fraction0.8670.875Tie
GenBench Human Enhancers CohnGenBench Human Enhancers CohnAccuracy (task-level value as printed in Table 4), fraction0.730.738Caduceus
GenBench Human Enhancers EnsemblGenBench Human Enhancers EnsemblAccuracy (task-level value as printed in Table 4), fraction0.9030.845HyenaDNA
GenBench Human Ensembl RegulatoryGenBench Human Ensembl RegulatoryAccuracy (task-level value as printed in Table 4), fraction0.8460.737HyenaDNA
GenBench Human Nontata PromotersGenBench Human Nontata PromotersAccuracy (task-level value as printed in Table 4), fraction0.9450.888HyenaDNA
GenBench Human OCR EnsemblGenBench Human OCR EnsemblAccuracy (task-level value as printed in Table 4), fraction0.7910.818Caduceus
GenBench Human Promoter DetectionGenBench Human Promoter DetectionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows), fraction0.730.732Tie
GenBench Human Promoter Detection, dataset allGenBench Human Promoter Detection, dataset allAccuracy, fraction0.7750.783Caduceus
GenBench Human Promoter Detection, dataset notataGenBench Human Promoter Detection, dataset notataAccuracy, fraction0.810.828Caduceus
GenBench Human Promoter Detection, dataset tataGenBench Human Promoter Detection, dataset tataAccuracy, fraction0.6040.585HyenaDNA
GenBench Human Splice Site DetectionGenBench Human Splice Site DetectionAccuracy (task-level value as printed in Table 4), fraction0.5660.567Tie
GenBench Human Transcription Factor PredictionGenBench Human Transcription Factor PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows), fraction0.6980.693Tie
GenBench Human Transcription Factor Prediction, dataset 0GenBench Human Transcription Factor Prediction, dataset 0Accuracy, fraction0.7220.714HyenaDNA
GenBench Human Transcription Factor Prediction, dataset 1GenBench Human Transcription Factor Prediction, dataset 1Accuracy, fraction0.740.759Caduceus
GenBench Human Transcription Factor Prediction, dataset 2GenBench Human Transcription Factor Prediction, dataset 2Accuracy, fraction0.6690.665Tie
GenBench Human Transcription Factor Prediction, dataset 3GenBench Human Transcription Factor Prediction, dataset 3Accuracy, fraction0.6430.637Tie
GenBench Human Transcription Factor Prediction, dataset 4GenBench Human Transcription Factor Prediction, dataset 4Accuracy, fraction0.7140.689HyenaDNA
GenBench Human vs WormGenBench Human vs WormAccuracy (task-level value as printed in Table 4), fraction0.9620.956Tie
GenBench Mouse EnhancersGenBench Mouse EnhancersAccuracy (task-level value as printed in Table 4), fraction0.7930.816Caduceus
GenBench Mouse Transcription Factor PredictionGenBench Mouse Transcription Factor PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows), fraction0.6530.652Tie
GenBench Mouse Transcription Factor Prediction, dataset 0GenBench Mouse Transcription Factor Prediction, dataset 0Accuracy, fraction0.6160.601HyenaDNA
GenBench Mouse Transcription Factor Prediction, dataset 1GenBench Mouse Transcription Factor Prediction, dataset 1Accuracy, fraction0.7280.739Caduceus
GenBench Mouse Transcription Factor Prediction, dataset 2GenBench Mouse Transcription Factor Prediction, dataset 2Accuracy, fraction0.7060.749Caduceus
GenBench Mouse Transcription Factor Prediction, dataset 3GenBench Mouse Transcription Factor Prediction, dataset 3Accuracy, fraction0.6210.579HyenaDNA
GenBench Mouse Transcription Factor Prediction, dataset 4GenBench Mouse Transcription Factor Prediction, dataset 4Accuracy, fraction0.5970.592Tie
GenBench Virus Covid Variant ClassificationGenBench Virus Covid Variant ClassificationAccuracy (task-level value as printed in Table 4), fraction0.3770.379Tie
GenBench Yeast Epigenetic Marks PredictionGenBench Yeast Epigenetic Marks PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows), fraction0.630.638Caduceus
GenBench Yeast Epigenetic Marks Prediction, dataset H3GenBench Yeast Epigenetic Marks Prediction, dataset H3Accuracy, fraction0.6850.715Caduceus
GenBench Yeast Epigenetic Marks Prediction, dataset H3K14acGenBench Yeast Epigenetic Marks Prediction, dataset H3K14acAccuracy, fraction0.6230.646Caduceus
GenBench Yeast Epigenetic Marks Prediction, dataset H3K36me3GenBench Yeast Epigenetic Marks Prediction, dataset H3K36me3Accuracy, fraction0.6330.653Caduceus
GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me1GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me1Accuracy, fraction0.5940.603Caduceus
GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me2GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me2Accuracy, fraction0.6090.594HyenaDNA
GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me3GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me3Accuracy, fraction0.5690.57Tie
GenBench Yeast Epigenetic Marks Prediction, dataset H3K79me3GenBench Yeast Epigenetic Marks Prediction, dataset H3K79me3Accuracy, fraction0.6590.659Tie
GenBench Yeast Epigenetic Marks Prediction, dataset H3K9acGenBench Yeast Epigenetic Marks Prediction, dataset H3K9acAccuracy, fraction0.6360.639Tie
GenBench Yeast Epigenetic Marks Prediction, dataset H4GenBench Yeast Epigenetic Marks Prediction, dataset H4Accuracy, fraction0.690.706Caduceus
GenBench Yeast Epigenetic Marks Prediction, dataset H4acGenBench Yeast Epigenetic Marks Prediction, dataset H4acAccuracy, fraction0.6030.595HyenaDNA
Comparisons in GENEB leaderboard (45)
ProtocolDatasetMetricHyenaDNACaduceusHigher
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Chromatin Acc.'), unitless0.340.337HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Coding/NC'), unitless0.5540.524HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'DNA Methyl.'), unitless0.05540.0447HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Enhancers'), unitless0.2640.193HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Histone Mod.'), unitless0.1980.143HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Mouse Enh.'), unitless0.04330.0275HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Promoters'), unitless0.4620.409HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Regulatory'), unitless0.1420.0879HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Species Clf.'), unitless0.3850.202HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Splice Sites'), unitless0.1250.173Caduceus
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'TF Binding'), unitless0.09210.0594HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Virus/Phage'), unitless0.1010.0292HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'lncRNA'), unitless0.02780.0425Caduceus
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; macro average: unweighted mean of the 13 category MCCs), unitless0.2150.175HyenaDNA
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; micro average: mean over all 100 tasks), unitless0.2290.188HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Chromatin Acc.'), unitless0.1290.215Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Coding/NC'), unitless0.3110.151HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'DNA Methyl.'), unitless0.006110.0117Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Enhancers'), unitless0.1330.0959HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Histone Mod.'), unitless0.08590.0802HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Mouse Enh.'), unitless0.01650.0234Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Promoters'), unitless0.2720.243HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Regulatory'), unitless0.05980.0477HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Species Clf.'), unitless0.180.0954HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Splice Sites'), unitless0.06440.0361HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'TF Binding'), unitless0.02980.0394Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Virus/Phage'), unitless0.04210.023HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'lncRNA'), unitless0.02130.0211HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; macro average: unweighted mean of the 13 category MCCs), unitless0.1040.0833HyenaDNA
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; micro average: mean over all 100 tasks), unitless0.1170.101HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Chromatin Acc.'), unitless0.540.197HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Coding/NC'), unitless0.7120.775Caduceus
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'DNA Methyl.'), unitless0.1690.0942HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Enhancers'), unitless0.4360.394HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Histone Mod.'), unitless0.4480.356HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Mouse Enh.'), unitless0.390.317HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Promoters'), unitless0.6960.603HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Regulatory'), unitless0.4230.303HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Species Clf.'), unitless0.6660.417HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Splice Sites'), unitless0.4010.334HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'TF Binding'), unitless0.5320.471HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Virus/Phage'), unitless0.3460.19HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'lncRNA'), unitless0.1580.173Caduceus
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; macro average: unweighted mean of the 13 category MCCs), unitless0.4550.356HyenaDNA
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; micro average: mean over all 100 tasks), unitless0.4680.384HyenaDNA
Comparisons in DNALONGBENCH (27)
ProtocolDatasetMetricHyenaDNACaduceusHigher
DNALONGBENCH Contact map prediction (Table 4)DNALONGBENCH Contact map prediction test setStratum adjusted correlation coefficient (GM12878), unitless0.0990.101Caduceus
DNALONGBENCH Contact map prediction (Table 4)DNALONGBENCH Contact map prediction test setStratum adjusted correlation coefficient (H1hESC), unitless0.1220.13Caduceus
DNALONGBENCH Contact map prediction (Table 4)DNALONGBENCH Contact map prediction test setStratum adjusted correlation coefficient (HCT116), unitless0.1180.145Caduceus
DNALONGBENCH Contact map prediction (Table 4)DNALONGBENCH Contact map prediction test setStratum adjusted correlation coefficient (HFF), unitless0.1390.153Caduceus
DNALONGBENCH Contact map prediction (Table 4)DNALONGBENCH Contact map prediction test setStratum adjusted correlation coefficient (IMR90), unitless0.0970.138Caduceus
DNALONGBENCH Contact map prediction (Table 4)DNALONGBENCH Contact map prediction test setStratum adjusted correlation coefficient (average over CMP conditions as printed), unitless0.1150.133Caduceus
DNALONGBENCH Expression quantitative trait loci prediction (Table 7)DNALONGBENCH Expression quantitative trait loci prediction test setAUROC (AS (Adipose_Subcutaneous)), unitless0.5130.541Caduceus
DNALONGBENCH Expression quantitative trait loci prediction (Table 7)DNALONGBENCH Expression quantitative trait loci prediction test setAUROC (AT (Artery_Tibial)), unitless0.4790.547Caduceus
DNALONGBENCH Expression quantitative trait loci prediction (Table 7)DNALONGBENCH Expression quantitative trait loci prediction test setAUROC (CCF (Cells_Cultured_fibroblasts)), unitless0.5840.597Caduceus
DNALONGBENCH Expression quantitative trait loci prediction (Table 7)DNALONGBENCH Expression quantitative trait loci prediction test setAUROC (MS (Muscle Skeletal)), unitless0.4870.538Caduceus
DNALONGBENCH Expression quantitative trait loci prediction (Table 7)DNALONGBENCH Expression quantitative trait loci prediction test setAUROC (NT (Nerve Tibial)), unitless0.5110.588Caduceus
DNALONGBENCH Expression quantitative trait loci prediction (Table 7)DNALONGBENCH Expression quantitative trait loci prediction test setAUROC (SNSES (Skin_Not_Sun_Exposed_Suprapubic)), unitless0.4710.586Caduceus
DNALONGBENCH Expression quantitative trait loci prediction (Table 7)DNALONGBENCH Expression quantitative trait loci prediction test setAUROC (SSELL (Skin_Sun_Exposed_Lower_leg)), unitless0.5440.574Caduceus
DNALONGBENCH Expression quantitative trait loci prediction (Table 7)DNALONGBENCH Expression quantitative trait loci prediction test setAUROC (Thyroid), unitless0.5290.527Tie
DNALONGBENCH Expression quantitative trait loci prediction (Table 7)DNALONGBENCH Expression quantitative trait loci prediction test setAUROC (WB (Whole_Blood)), unitless0.5120.594Caduceus
DNALONGBENCH Expression quantitative trait loci prediction (Table 7)DNALONGBENCH Expression quantitative trait loci prediction test setAUROC (average over eQTLP conditions as printed), unitless0.5140.565Caduceus
DNALONGBENCH Enhancer-target gene prediction (Table 3)DNALONGBENCH Enhancer-target gene prediction test setAUPRC, unitless0.2490.38Caduceus
DNALONGBENCH Enhancer-target gene prediction (Table 3)DNALONGBENCH Enhancer-target gene prediction test setAUROC, unitless0.8280.826Tie
DNALONGBENCH Regulatory sequence activity prediction (Table 5)DNALONGBENCH Regulatory sequence activity prediction test setPearson (Human), unitless0.2980.301Tie
DNALONGBENCH Regulatory sequence activity prediction (Table 5)DNALONGBENCH Regulatory sequence activity prediction test setPearson (Mouse), unitless0.3960.4Tie
DNALONGBENCH Regulatory sequence activity prediction (Table 5)DNALONGBENCH Regulatory sequence activity prediction test setPearson (average over RSAP conditions as printed), unitless0.3470.349Tie
DNALONGBENCH Transcription initiation signal prediction (Table 6)DNALONGBENCH Transcription initiation signal prediction test setPearson (EC (ENCODE CAGE)), unitless0.1240.088HyenaDNA
DNALONGBENCH Transcription initiation signal prediction (Table 6)DNALONGBENCH Transcription initiation signal prediction test setPearson (ER (ENCODE RAMPAGE)), unitless0.1180.088HyenaDNA
DNALONGBENCH Transcription initiation signal prediction (Table 6)DNALONGBENCH Transcription initiation signal prediction test setPearson (FC (FANTOM CAGE)), unitless0.1380.114HyenaDNA
DNALONGBENCH Transcription initiation signal prediction (Table 6)DNALONGBENCH Transcription initiation signal prediction test setPearson (GC (GRO-cap)), unitless0.1120.097HyenaDNA
DNALONGBENCH Transcription initiation signal prediction (Table 6)DNALONGBENCH Transcription initiation signal prediction test setPearson (PC (PRO-cap)), unitless0.1680.154HyenaDNA
DNALONGBENCH Transcription initiation signal prediction (Table 6)DNALONGBENCH Transcription initiation signal prediction test setPearson (average over TISP conditions as printed), unitless0.1320.109HyenaDNA
Comparisons in Comparative Analysis of Deep Learning… (4)
ProtocolDatasetMetricHyenaDNACaduceusHigher
Allelic reporter effect correlation in HeLa (Manzo et al. 2025 Table 1)HeLa regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.0550.02HyenaDNA
Allelic reporter effect correlation in HepG2 (Manzo et al. 2025 Table 1)HepG2 regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.1340.108HyenaDNA
Allelic reporter effect correlation in K562 (Manzo et al. 2025 Table 1)K562 regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.0770.133Caduceus
Allelic reporter effect correlation in NPC (Manzo et al. 2025 Table 1)NPC regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless-0.0070.023Caduceus
Comparisons in TraitGym official metrics (2)
ProtocolDatasetMetricHyenaDNACaduceusHigher
TraitGym Complex traits, zero-shot, AUPRC by chromosome weighted averageTraitGym Complex traits, matched 9 controlsAUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all'), unitless0.1110.0971HyenaDNA
TraitGym Mendelian traits, zero-shot, AUPRC by chromosome weighted averageTraitGym Mendelian traits, matched 9 controlsAUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all'), unitless0.1060.0878HyenaDNA

What disagrees

Other studies
rank HyenaDNA above Caduceus in 39 of 47 comparisons
This study
ranks Caduceus above HyenaDNA in 9 of 12 comparisons
Compared
Caduceus, HyenaDNA
Study that disagrees
A Phylogenetic Approach to Genomic Language Modeling, Springer chapter page for the RECOMB 2025 version of record, Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)
Other studies
GENEB leaderboard (leaderboard.json, 13 task-type aggregates), TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym), Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)
Rule
Ordering reversal; unknown
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (24)
StudyToolProtocol or benchmarkMetricValue
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Downstream of Transcript)0.72
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Downstream of Transcript)0.55
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Intronic)0.64
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Intronic)0.48
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Missense)0.55
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Missense)0.54
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Noncoding)0.59
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Noncoding)0.47
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Acceptor)0.61
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Acceptor)0.51
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Donor)0.68
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Donor)0.52
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Start Codon)0.41
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Start Codon)0.43
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Stop Lost)0.43
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Stop Lost)0.48
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Synonymous)0.55
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Synonymous)0.43
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Upstream of Transcript)0.4
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Upstream of Transcript)0.51
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 3’ UTR)0.63
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 3’ UTR)0.48
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)Caduceus-PS 131k, 256 model dimensions, 16 layersClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 5’ UTR)0.61
A Phylogenetic Approach to Genomic Language Modeling; Supplement 1 of the PhyloGPN preprint (arXiv:2503.03773v1)HyenaDNA medium 160k, likelihood-ratio scoringClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 5’ UTR)0.48
In the other studies (94 results)
In the other studies
StudyToolProtocol or benchmarkMetricValue
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Chromatin Acc.')0.337
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Coding/NC')0.524
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'DNA Methyl.')0.0447
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Enhancers')0.193
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Histone Mod.')0.143
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'lncRNA')0.0425
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; macro average: unweighted mean of the 13 category MCCs)0.175
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; micro average: mean over all 100 tasks)0.188
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Mouse Enh.')0.0275
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Promoters')0.409
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Regulatory')0.0879
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Species Clf.')0.202
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Splice Sites')0.173
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'TF Binding')0.0594
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Virus/Phage')0.0292
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Chromatin Acc.')0.215
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Coding/NC')0.151
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'DNA Methyl.')0.0117
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Enhancers')0.0959
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Histone Mod.')0.0802
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'lncRNA')0.0211
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; macro average: unweighted mean of the 13 category MCCs)0.0833
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; micro average: mean over all 100 tasks)0.101
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Mouse Enh.')0.0234
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Promoters')0.243
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Regulatory')0.0477
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Species Clf.')0.0954
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Splice Sites')0.0361
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'TF Binding')0.0394
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Virus/Phage')0.023
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Chromatin Acc.')0.197
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Coding/NC')0.775
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'DNA Methyl.')0.0942
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Enhancers')0.394
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Histone Mod.')0.356
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'lncRNA')0.173
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; macro average: unweighted mean of the 13 category MCCs)0.356
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; micro average: mean over all 100 tasks)0.384
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Mouse Enh.')0.317
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Promoters')0.603
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Regulatory')0.303
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Species Clf.')0.417
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Splice Sites')0.334
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'TF Binding')0.471
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)Caduceus-PS-131k-256 (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Virus/Phage')0.19
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Chromatin Acc.')0.34
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Coding/NC')0.554
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'DNA Methyl.')0.0554
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Enhancers')0.264
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Histone Mod.')0.198
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'lncRNA')0.0278
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; macro average: unweighted mean of the 13 category MCCs)0.215
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; micro average: mean over all 100 tasks)0.229
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Mouse Enh.')0.0433
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Promoters')0.462
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Regulatory')0.142
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Species Clf.')0.385
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Splice Sites')0.125
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'TF Binding')0.0921
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 10-shot, MCC aggregatesMCC (10-shot; category 'Virus/Phage')0.101
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Chromatin Acc.')0.129
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Coding/NC')0.311
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'DNA Methyl.')0.00611
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Enhancers')0.133
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Histone Mod.')0.0859
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'lncRNA')0.0213
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; macro average: unweighted mean of the 13 category MCCs)0.104
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; micro average: mean over all 100 tasks)0.117
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Mouse Enh.')0.0165
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Promoters')0.272
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Regulatory')0.0598
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Species Clf.')0.18
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Splice Sites')0.0644
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'TF Binding')0.0298
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, 1-shot, MCC aggregatesMCC (1-shot; category 'Virus/Phage')0.0421
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Chromatin Acc.')0.54
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Coding/NC')0.712
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'DNA Methyl.')0.169
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Enhancers')0.436
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Histone Mod.')0.448
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'lncRNA')0.158
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; macro average: unweighted mean of the 13 category MCCs)0.455
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; micro average: mean over all 100 tasks)0.468
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Mouse Enh.')0.39
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Promoters')0.696
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Regulatory')0.423
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Species Clf.')0.666
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Splice Sites')0.401
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'TF Binding')0.532
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)hyenadna-large-1m (GENEB leaderboard)GENEB leaderboard, full training data, MCC aggregatesMCC (full training data; category 'Virus/Phage')0.346
TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)Caduceus_absLLR.plus.score (TraitGym)TraitGym Complex traits, zero-shot, AUPRC by chromosome weighted averageAUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all')0.0971
TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)Caduceus_absLLR.plus.score (TraitGym)TraitGym Mendelian traits, zero-shot, AUPRC by chromosome weighted averageAUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all')0.0878
TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)HyenaDNA_absLLR.plus.score (TraitGym)TraitGym Complex traits, zero-shot, AUPRC by chromosome weighted averageAUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all')0.111
TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)HyenaDNA_absLLR.plus.score (TraitGym)TraitGym Mendelian traits, zero-shot, AUPRC by chromosome weighted averageAUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all')0.106

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-8d690be6eb693120 · All contested results · How findings are made and followed up