rewirebio.iobenchmarks
Contested result

Caduceus and GENA-LM: GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models ranks GENA-LM above Caduceus in 41 of 48 comparisons; 2 of 2 other studies rank Caduceus above GENA-LM.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

3 papers rank Caduceus above GENA-LM in 23 of their 55 comparisons and GENA-LM higher in 31; 1 is a tie. Of those papers, GENEB leaderboard ranks GENA-LM higher on its own (30 of 45). GenBench ranks GENA-LM above Caduceus in 41 of its 51 comparisons and Caduceus higher in 7; 3 are ties.

Papers grouped by which system they rank higher: Caduceus or GENA-LMRank Caduceus higher: GUANinE v1.1 reveals complementarity…, Comparative Analysis of Deep Learning…. Rank GENA-LM higher: GenBench (disagrees), GENEB leaderboard. Mixed: none.Rank Caduceus higherMixedRank GENA-LM higherGUANinE v1.1 revealscomplementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language modelsComparative Analysisof Deep Learning…Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsGenBenchGenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)
  • Rank Caduceus higher (2): GUANinE v1.1 reveals complementarity…, Comparative Analysis of Deep Learning…
  • Rank GENA-LM higher (2): GenBench (disagrees), GENEB leaderboard
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Caduceus or GENA-LM higherGenBench (disagrees): Caduceus higher in 7, GENA-LM higher in 41, 3 ties. GENEB leaderboard: Caduceus higher in 15, GENA-LM higher in 30, 0 ties. GUANinE v1.1 reveals complementarity…: Caduceus higher in 5, GENA-LM higher in 0, 1 tie. Comparative Analysis of Deep Learning…: Caduceus higher in 3, GENA-LM higher in 1, 0 ties.Disagrees: GenBenchGenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models7 Caduceus higher73 tie41 GENA-LM higher41GENEB leaderboardGENEB leaderboard (leaderboard.json, 13 task-type aggregates)15 Caduceus higher1530 GENA-LM higher30GUANinE v1.1 reveals complementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language models5 Caduceus higher51 tie1Comparative Analysis of Deep Learning…Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants3 Caduceus higher31 GENA-LM higher1
Caduceus higherTieGENA-LM higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in GenBench (51)
ProtocolDatasetMetricCaduceusGENA-LMHigher
GenBench Bulk RNA expressionGenBench Bulk RNA expressionMSE, unitless0.510.496GENA-LM
GenBench Bulk RNA expressionGenBench Bulk RNA expressionSpearman, unitless0.7460.748Tie
GenBench Drosophila enhancer activity (dev)GenBench Drosophila enhancer activity (dev)Pearson, unitless0.4430.624GENA-LM
GenBench Drosophila enhancer activity (hk)GenBench Drosophila enhancer activity (hk)Pearson, unitless0.530.74GENA-LM
GenBench Drosophila enhancer activity (Mean)GenBench Drosophila enhancer activity (Mean)Pearson (mean of dev and hk as printed), unitless0.4860.682GENA-LM
GenBench Long-range Promoters PredictionGenBench Long-range Promoters PredictionAccuracy, fraction0.930.98GENA-LM
GenBench Long-range Species ClassificationGenBench Long-range Species ClassificationAccuracy, fraction0.7110.773GENA-LM
GenBench Long-range splice site prediction (Mean)GenBench Long-range splice site prediction (Mean)AUROC (mean of donor and acceptor as printed), unitless0.6910.679Caduceus
GenBench Long-range splice site prediction (Splice acceptor)GenBench Long-range splice site prediction (Splice acceptor)AUROC, unitless0.740.73Caduceus
GenBench Long-range splice site prediction (Splice donar)GenBench Long-range splice site prediction (Splice donar)AUROC, unitless0.6420.629Caduceus
GenBench Coding vs IntergenomicGenBench Coding vs IntergenomicAccuracy (task-level value as printed in Table 4), fraction0.9370.932Tie
GenBench Human Core Promoter DetectionGenBench Human Core Promoter DetectionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows), fraction0.8510.814Caduceus
GenBench Human Core Promoter Detection, dataset allGenBench Human Core Promoter Detection, dataset allAccuracy, fraction0.8340.816Caduceus
GenBench Human Core Promoter Detection, dataset notataGenBench Human Core Promoter Detection, dataset notataAccuracy, fraction0.8430.836Tie
GenBench Human Core Promoter Detection, dataset tataGenBench Human Core Promoter Detection, dataset tataAccuracy, fraction0.8750.791Caduceus
GenBench Human Enhancers CohnGenBench Human Enhancers CohnAccuracy (task-level value as printed in Table 4), fraction0.7380.756GENA-LM
GenBench Human Enhancers EnsemblGenBench Human Enhancers EnsemblAccuracy (task-level value as printed in Table 4), fraction0.8450.911GENA-LM
GenBench Human Ensembl RegulatoryGenBench Human Ensembl RegulatoryAccuracy (task-level value as printed in Table 4), fraction0.7370.881GENA-LM
GenBench Human Nontata PromotersGenBench Human Nontata PromotersAccuracy (task-level value as printed in Table 4), fraction0.8880.966GENA-LM
GenBench Human OCR EnsemblGenBench Human OCR EnsemblAccuracy (task-level value as printed in Table 4), fraction0.8180.79Caduceus
GenBench Human Promoter DetectionGenBench Human Promoter DetectionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows), fraction0.7320.9GENA-LM
GenBench Human Promoter Detection, dataset allGenBench Human Promoter Detection, dataset allAccuracy, fraction0.7830.929GENA-LM
GenBench Human Promoter Detection, dataset notataGenBench Human Promoter Detection, dataset notataAccuracy, fraction0.8280.962GENA-LM
GenBench Human Promoter Detection, dataset tataGenBench Human Promoter Detection, dataset tataAccuracy, fraction0.5850.81GENA-LM
GenBench Human Splice Site DetectionGenBench Human Splice Site DetectionAccuracy (task-level value as printed in Table 4), fraction0.5670.918GENA-LM
GenBench Human Transcription Factor PredictionGenBench Human Transcription Factor PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows), fraction0.6930.824GENA-LM
GenBench Human Transcription Factor Prediction, dataset 0GenBench Human Transcription Factor Prediction, dataset 0Accuracy, fraction0.7140.832GENA-LM
GenBench Human Transcription Factor Prediction, dataset 1GenBench Human Transcription Factor Prediction, dataset 1Accuracy, fraction0.7590.851GENA-LM
GenBench Human Transcription Factor Prediction, dataset 2GenBench Human Transcription Factor Prediction, dataset 2Accuracy, fraction0.6650.812GENA-LM
GenBench Human Transcription Factor Prediction, dataset 3GenBench Human Transcription Factor Prediction, dataset 3Accuracy, fraction0.6370.777GENA-LM
GenBench Human Transcription Factor Prediction, dataset 4GenBench Human Transcription Factor Prediction, dataset 4Accuracy, fraction0.6890.855GENA-LM
GenBench Human vs WormGenBench Human vs WormAccuracy (task-level value as printed in Table 4), fraction0.9560.97GENA-LM
GenBench Mouse EnhancersGenBench Mouse EnhancersAccuracy (task-level value as printed in Table 4), fraction0.8160.83GENA-LM
GenBench Mouse Transcription Factor PredictionGenBench Mouse Transcription Factor PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows), fraction0.6520.827GENA-LM
GenBench Mouse Transcription Factor Prediction, dataset 0GenBench Mouse Transcription Factor Prediction, dataset 0Accuracy, fraction0.6010.757GENA-LM
GenBench Mouse Transcription Factor Prediction, dataset 1GenBench Mouse Transcription Factor Prediction, dataset 1Accuracy, fraction0.7390.908GENA-LM
GenBench Mouse Transcription Factor Prediction, dataset 2GenBench Mouse Transcription Factor Prediction, dataset 2Accuracy, fraction0.7490.905GENA-LM
GenBench Mouse Transcription Factor Prediction, dataset 3GenBench Mouse Transcription Factor Prediction, dataset 3Accuracy, fraction0.5790.829GENA-LM
GenBench Mouse Transcription Factor Prediction, dataset 4GenBench Mouse Transcription Factor Prediction, dataset 4Accuracy, fraction0.5920.733GENA-LM
GenBench Virus Covid Variant ClassificationGenBench Virus Covid Variant ClassificationAccuracy (task-level value as printed in Table 4), fraction0.3790.703GENA-LM
GenBench Yeast Epigenetic Marks PredictionGenBench Yeast Epigenetic Marks PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows), fraction0.6380.783GENA-LM
GenBench Yeast Epigenetic Marks Prediction, dataset H3GenBench Yeast Epigenetic Marks Prediction, dataset H3Accuracy, fraction0.7150.889GENA-LM
GenBench Yeast Epigenetic Marks Prediction, dataset H3K14acGenBench Yeast Epigenetic Marks Prediction, dataset H3K14acAccuracy, fraction0.6460.76GENA-LM
GenBench Yeast Epigenetic Marks Prediction, dataset H3K36me3GenBench Yeast Epigenetic Marks Prediction, dataset H3K36me3Accuracy, fraction0.6530.776GENA-LM
GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me1GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me1Accuracy, fraction0.6030.721GENA-LM
GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me2GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me2Accuracy, fraction0.5940.707GENA-LM
GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me3GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me3Accuracy, fraction0.570.714GENA-LM
GenBench Yeast Epigenetic Marks Prediction, dataset H3K79me3GenBench Yeast Epigenetic Marks Prediction, dataset H3K79me3Accuracy, fraction0.6590.832GENA-LM
GenBench Yeast Epigenetic Marks Prediction, dataset H3K9acGenBench Yeast Epigenetic Marks Prediction, dataset H3K9acAccuracy, fraction0.6390.784GENA-LM
GenBench Yeast Epigenetic Marks Prediction, dataset H4GenBench Yeast Epigenetic Marks Prediction, dataset H4Accuracy, fraction0.7060.892GENA-LM
GenBench Yeast Epigenetic Marks Prediction, dataset H4acGenBench Yeast Epigenetic Marks Prediction, dataset H4acAccuracy, fraction0.5950.753GENA-LM
Comparisons in GENEB leaderboard (45)
ProtocolDatasetMetricCaduceusGENA-LMHigher
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Chromatin Acc.'), unitless0.3370.499GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Coding/NC'), unitless0.5240.626GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'DNA Methyl.'), unitless0.04470.0625GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Enhancers'), unitless0.1930.272GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Histone Mod.'), unitless0.1430.181GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Mouse Enh.'), unitless0.02750.152GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Promoters'), unitless0.4090.49GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Regulatory'), unitless0.08790.151GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Species Clf.'), unitless0.2020.482GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Splice Sites'), unitless0.1730.158Caduceus
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'TF Binding'), unitless0.05940.223GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'Virus/Phage'), unitless0.02920.118GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; category 'lncRNA'), unitless0.04250.027Caduceus
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; macro average: unweighted mean of the 13 category MCCs), unitless0.1750.265GENA-LM
GENEB leaderboard, 10-shot, MCC aggregatesGENEB 100 tasksMCC (10-shot; micro average: mean over all 100 tasks), unitless0.1880.251GENA-LM
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Chromatin Acc.'), unitless0.2150.0442Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Coding/NC'), unitless0.1510.0399Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'DNA Methyl.'), unitless0.01170.00111Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Enhancers'), unitless0.09590.0616Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Histone Mod.'), unitless0.08020.0383Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Mouse Enh.'), unitless0.02340.00967Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Promoters'), unitless0.2430.11Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Regulatory'), unitless0.04770.0356Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Species Clf.'), unitless0.09540.117GENA-LM
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Splice Sites'), unitless0.03610.0517GENA-LM
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'TF Binding'), unitless0.03940.0292Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'Virus/Phage'), unitless0.0230.0121Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; category 'lncRNA'), unitless0.02110.0182Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; macro average: unweighted mean of the 13 category MCCs), unitless0.08330.0437Caduceus
GENEB leaderboard, 1-shot, MCC aggregatesGENEB 100 tasksMCC (1-shot; micro average: mean over all 100 tasks), unitless0.1010.0526Caduceus
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Chromatin Acc.'), unitless0.1970.461GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Coding/NC'), unitless0.7750.83GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'DNA Methyl.'), unitless0.09420.198GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Enhancers'), unitless0.3940.477GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Histone Mod.'), unitless0.3560.449GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Mouse Enh.'), unitless0.3170.48GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Promoters'), unitless0.6030.697GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Regulatory'), unitless0.3030.449GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Species Clf.'), unitless0.4170.674GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Splice Sites'), unitless0.3340.42GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'TF Binding'), unitless0.4710.59GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'Virus/Phage'), unitless0.190.481GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; category 'lncRNA'), unitless0.1730.249GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; macro average: unweighted mean of the 13 category MCCs), unitless0.3560.496GENA-LM
GENEB leaderboard, full training data, MCC aggregatesGENEB 100 tasksMCC (full training data; micro average: mean over all 100 tasks), unitless0.3840.492GENA-LM
Comparisons in GUANinE v1.1 reveals complementarity… (6)
Comparisons in Comparative Analysis of Deep Learning… (4)
ProtocolDatasetMetricCaduceusGENA-LMHigher
Allelic reporter effect correlation in HeLa (Manzo et al. 2025 Table 1)HeLa regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.020.026GENA-LM
Allelic reporter effect correlation in HepG2 (Manzo et al. 2025 Table 1)HepG2 regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.1080.062Caduceus
Allelic reporter effect correlation in K562 (Manzo et al. 2025 Table 1)K562 regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.1330.077Caduceus
Allelic reporter effect correlation in NPC (Manzo et al. 2025 Table 1)NPC regulatory variant reporter data (Manzo et al. 2025)Pearson (predicted versus measured log2 fold-change of variant effect), unitless0.0230.01Caduceus

What disagrees

Other studies
rank Caduceus above GENA-LM in 8 of 9 comparisons
This study
ranks GENA-LM above Caduceus in 41 of 48 comparisons
Compared
Caduceus, GENA-LM
Study that disagrees
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models
Other studies
GUANinE v1.1 reveals complementarity of supervised and genomic language models, Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory Variants
Rule
Ordering reversal; unknown
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (96)
StudyToolProtocol or benchmarkMetricValue
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Bulk RNA expressionMSE0.51
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Drosophila enhancer activity (dev)Pearson0.443
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Drosophila enhancer activity (hk)Pearson0.53
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Drosophila enhancer activity (Mean)Pearson (mean of dev and hk as printed)0.486
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Long-range Promoters PredictionAccuracy0.93
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Long-range Species ClassificationAccuracy0.711
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Long-range splice site prediction (Mean)AUROC (mean of donor and acceptor as printed)0.691
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Long-range splice site prediction (Splice acceptor)AUROC0.74
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Long-range splice site prediction (Splice donar)AUROC0.642
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Core Promoter DetectionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows)0.851
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Core Promoter Detection, dataset allAccuracy0.834
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Core Promoter Detection, dataset tataAccuracy0.875
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Enhancers CohnAccuracy (task-level value as printed in Table 4)0.738
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Enhancers EnsemblAccuracy (task-level value as printed in Table 4)0.845
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Ensembl RegulatoryAccuracy (task-level value as printed in Table 4)0.737
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Nontata PromotersAccuracy (task-level value as printed in Table 4)0.888
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human OCR EnsemblAccuracy (task-level value as printed in Table 4)0.818
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Promoter DetectionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows)0.732
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Promoter Detection, dataset allAccuracy0.783
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Promoter Detection, dataset notataAccuracy0.828
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Promoter Detection, dataset tataAccuracy0.585
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Splice Site DetectionAccuracy (task-level value as printed in Table 4)0.567
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Transcription Factor PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows)0.693
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Transcription Factor Prediction, dataset 0Accuracy0.714
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Transcription Factor Prediction, dataset 1Accuracy0.759
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Transcription Factor Prediction, dataset 2Accuracy0.665
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Transcription Factor Prediction, dataset 3Accuracy0.637
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human Transcription Factor Prediction, dataset 4Accuracy0.689
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Human vs WormAccuracy (task-level value as printed in Table 4)0.956
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Mouse EnhancersAccuracy (task-level value as printed in Table 4)0.816
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Mouse Transcription Factor PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows)0.652
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Mouse Transcription Factor Prediction, dataset 0Accuracy0.601
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Mouse Transcription Factor Prediction, dataset 1Accuracy0.739
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Mouse Transcription Factor Prediction, dataset 2Accuracy0.749
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Mouse Transcription Factor Prediction, dataset 3Accuracy0.579
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Mouse Transcription Factor Prediction, dataset 4Accuracy0.592
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Virus Covid Variant ClassificationAccuracy (task-level value as printed in Table 4)0.379
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows)0.638
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3Accuracy0.715
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K14acAccuracy0.646
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K36me3Accuracy0.653
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me1Accuracy0.603
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me2Accuracy0.594
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me3Accuracy0.57
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K79me3Accuracy0.659
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K9acAccuracy0.639
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H4Accuracy0.706
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsCaduceus (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H4acAccuracy0.595
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Bulk RNA expressionMSE0.496
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Drosophila enhancer activity (dev)Pearson0.624
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Drosophila enhancer activity (hk)Pearson0.74
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Drosophila enhancer activity (Mean)Pearson (mean of dev and hk as printed)0.682
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Long-range Promoters PredictionAccuracy0.98
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Long-range Species ClassificationAccuracy0.773
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Long-range splice site prediction (Mean)AUROC (mean of donor and acceptor as printed)0.679
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Long-range splice site prediction (Splice acceptor)AUROC0.73
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Long-range splice site prediction (Splice donar)AUROC0.629
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Core Promoter DetectionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows)0.814
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Core Promoter Detection, dataset allAccuracy0.816
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Core Promoter Detection, dataset tataAccuracy0.791
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Enhancers CohnAccuracy (task-level value as printed in Table 4)0.756
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Enhancers EnsemblAccuracy (task-level value as printed in Table 4)0.911
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Ensembl RegulatoryAccuracy (task-level value as printed in Table 4)0.881
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Nontata PromotersAccuracy (task-level value as printed in Table 4)0.966
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human OCR EnsemblAccuracy (task-level value as printed in Table 4)0.79
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Promoter DetectionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows)0.9
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Promoter Detection, dataset allAccuracy0.929
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Promoter Detection, dataset notataAccuracy0.962
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Promoter Detection, dataset tataAccuracy0.81
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Splice Site DetectionAccuracy (task-level value as printed in Table 4)0.918
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Transcription Factor PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows)0.824
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Transcription Factor Prediction, dataset 0Accuracy0.832
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Transcription Factor Prediction, dataset 1Accuracy0.851
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Transcription Factor Prediction, dataset 2Accuracy0.812
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Transcription Factor Prediction, dataset 3Accuracy0.777
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human Transcription Factor Prediction, dataset 4Accuracy0.855
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Human vs WormAccuracy (task-level value as printed in Table 4)0.97
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Mouse EnhancersAccuracy (task-level value as printed in Table 4)0.83
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Mouse Transcription Factor PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows)0.827
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Mouse Transcription Factor Prediction, dataset 0Accuracy0.757
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Mouse Transcription Factor Prediction, dataset 1Accuracy0.908
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Mouse Transcription Factor Prediction, dataset 2Accuracy0.905
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Mouse Transcription Factor Prediction, dataset 3Accuracy0.829
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Mouse Transcription Factor Prediction, dataset 4Accuracy0.733
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Virus Covid Variant ClassificationAccuracy (task-level value as printed in Table 4)0.703
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks PredictionAccuracy (task-level value as printed in Table 4; mean of the Table 10 dataset rows)0.783
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3Accuracy0.889
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K14acAccuracy0.76
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K36me3Accuracy0.776
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me1Accuracy0.721
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me2Accuracy0.707
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K4me3Accuracy0.714
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K79me3Accuracy0.832
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H3K9acAccuracy0.784
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H4Accuracy0.892
GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation ModelsGENA-LM (GenBench)GenBench Yeast Epigenetic Marks Prediction, dataset H4acAccuracy0.753
In the other studies (18)
StudyToolProtocol or benchmarkMetricValue
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsCaduceus-PS (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 ccre-propensity, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)41.3
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsCaduceus-PS (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 cons100, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)62.4
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsCaduceus-PS (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 cons30, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)74.4
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsCaduceus-PS (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 gpra-c, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)72.4
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsCaduceus-PS (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 gpra-d, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)76
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsGENA-LM-bert-base-t2t (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 ccre-propensity, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)34.9
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsGENA-LM-bert-base-t2t (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 cons100, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)56
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsGENA-LM-bert-base-t2t (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 cons30, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)66.7
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsGENA-LM-bert-base-t2t (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 gpra-c, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)71.1
GUANinE v1.1 reveals complementarity of supervised and genomic language modelsGENA-LM-bert-base-t2t (1% eval (evaluated with 1% of the training set), GUANinE v1.1)GUANinE v1.1 gpra-d, Spearman rho x100 (Table 2)Spearman (GUANinE v1.1 test set)74.8
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsCaduceus (Manzo et al. 2025)Allelic reporter effect correlation in HeLa (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.02
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsGena LM b-multi (Manzo et al. 2025)Allelic reporter effect correlation in HeLa (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.026
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsCaduceus (Manzo et al. 2025)Allelic reporter effect correlation in HepG2 (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.108
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsGena LM b-multi (Manzo et al. 2025)Allelic reporter effect correlation in HepG2 (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.062
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsCaduceus (Manzo et al. 2025)Allelic reporter effect correlation in K562 (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.133
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsGena LM b-multi (Manzo et al. 2025)Allelic reporter effect correlation in K562 (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.077
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsCaduceus (Manzo et al. 2025)Allelic reporter effect correlation in NPC (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.023
Comparative Analysis of Deep Learning Models for Predicting Causative Regulatory VariantsGena LM b-multi (Manzo et al. 2025)Allelic reporter effect correlation in NPC (Manzo et al. 2025 Table 1)Pearson (predicted versus measured log2 fold-change of variant effect)0.01

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-2bed2b1311dad57e · All contested results · How findings are made and followed up