Contested result

Nucleotide Transformer v2 and Nucleotide Transformer: GENEB leaderboard (leaderboard.json, 13 task-type aggregates) reverses the order

Low severity
Not yet reviewed
Ordering reversal
different-task

GENEB leaderboard (leaderboard.json, 13 task-type aggregates) ranks Nucleotide Transformer above Nucleotide Transformer v2 in 37 of 44 comparisons; 2 of 2 other studies rank Nucleotide Transformer v2 above Nucleotide Transformer.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank Nucleotide Transformer v2 above Nucleotide Transformer in 33 of their 46 comparisons and Nucleotide Transformer higher in 7; 6 are ties. GENEB leaderboard ranks Nucleotide Transformer above Nucleotide Transformer v2 in 37 of its 45 comparisons and Nucleotide Transformer v2 higher in 7; 1 is a tie.

Rank Nucleotide Transformer v2 higher (2): NABench, Nucleotide Transformer
Rank Nucleotide Transformer higher (1): GENEB leaderboard (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Nucleotide Transformer v2 higherTieNucleotide Transformer higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (3)
Paper	Role	Ranks Nucleotide Transformer v2 higher	Ranks Nucleotide Transformer higher	Ties
GENEB leaderboard (leaderboard.json, 13 task-type aggregates) · Repository	Disagrees	7	37	1
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction · Source	Agrees	19	6	2
Nucleotide Transformer: building and evaluating robust foundation models for human genomics (Nature Methods) and 1 more record · DOI	Agrees	14	1	4

Comparisons in GENEB leaderboard (45)

Protocol	Dataset	Metric	Nucleotide Transformer v2	Nucleotide Transformer	Higher
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'Chromatin Acc.'), unitless	0.386	0.41	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'Coding/NC'), unitless	0.534	0.667	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'DNA Methyl.'), unitless	0.0411	0.0575	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'Enhancers'), unitless	0.288	0.314	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'Histone Mod.'), unitless	0.261	0.258	Nucleotide Transformer v2
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'Mouse Enh.'), unitless	0.121	0.202	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'Promoters'), unitless	0.482	0.532	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'Regulatory'), unitless	0.178	0.152	Nucleotide Transformer v2
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'Species Clf.'), unitless	0.52	0.512	Nucleotide Transformer v2
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'Splice Sites'), unitless	0.0802	0.242	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'TF Binding'), unitless	0.125	0.201	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'Virus/Phage'), unitless	0.144	0.161	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; category 'lncRNA'), unitless	0.0528	0.103	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; macro average: unweighted mean of the 13 category MCCs), unitless	0.247	0.293	Nucleotide Transformer
GENEB leaderboard, 10-shot, MCC aggregates	GENEB 100 tasks	MCC (10-shot; micro average: mean over all 100 tasks), unitless	0.263	0.3	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'Chromatin Acc.'), unitless	0.176	0.169	Nucleotide Transformer v2
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'Coding/NC'), unitless	0.284	0.251	Nucleotide Transformer v2
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'DNA Methyl.'), unitless	0.0159	0.0179	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'Enhancers'), unitless	0.159	0.171	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'Histone Mod.'), unitless	0.0829	0.0952	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'Mouse Enh.'), unitless	0.0248	0.0481	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'Promoters'), unitless	0.222	0.309	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'Regulatory'), unitless	0.0692	0.0837	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'Species Clf.'), unitless	0.21	0.192	Nucleotide Transformer v2
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'Splice Sites'), unitless	0.0478	0.0768	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'TF Binding'), unitless	0.0202	0.0361	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'Virus/Phage'), unitless	0.0062	0.0194	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; category 'lncRNA'), unitless	0.00862	0.0454	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; macro average: unweighted mean of the 13 category MCCs), unitless	0.102	0.116	Nucleotide Transformer
GENEB leaderboard, 1-shot, MCC aggregates	GENEB 100 tasks	MCC (1-shot; micro average: mean over all 100 tasks), unitless	0.106	0.136	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'Chromatin Acc.'), unitless	0.652	0.554	Nucleotide Transformer v2
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'Coding/NC'), unitless	0.812	0.864	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'DNA Methyl.'), unitless	0.339	0.356	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'Enhancers'), unitless	0.469	0.487	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'Histone Mod.'), unitless	0.468	0.498	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'Mouse Enh.'), unitless	0.486	0.547	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'Promoters'), unitless	0.722	0.739	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'Regulatory'), unitless	0.457	0.488	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'Species Clf.'), unitless	0.747	0.752	Tie
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'Splice Sites'), unitless	0.47	0.652	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'TF Binding'), unitless	0.537	0.573	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'Virus/Phage'), unitless	0.482	0.564	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; category 'lncRNA'), unitless	0.337	0.372	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; macro average: unweighted mean of the 13 category MCCs), unitless	0.537	0.573	Nucleotide Transformer
GENEB leaderboard, full training data, MCC aggregates	GENEB 100 tasks	MCC (full training data; micro average: mean over all 100 tasks), unitless	0.524	0.561	Nucleotide Transformer

Comparisons in NABench (27)

Protocol	Dataset	Metric	Nucleotide Transformer v2	Nucleotide Transformer	Higher
NABench FS-CORR-APTAMER: Fitness prediction on aptamer assays, few-shot	NABench aptamer assays (NABench split)	Spearman, unitless	0.23	0.166	Nucleotide Transformer v2
NABench FS-CORR-ENHANCER: Fitness prediction on enhancer assays, few-shot	NABench enhancer assays (NABench split)	Spearman, unitless	0.026	0.053	Nucleotide Transformer
NABench FS-CORR-MRNA: Fitness prediction on mRNA assays, few-shot	NABench mRNA assays (NABench split)	Spearman, unitless	0.214	0.256	Nucleotide Transformer
NABench FS-CORR-PROMOTER: Fitness prediction on promoter assays, few-shot	NABench promoter assays (NABench split)	Spearman, unitless	0.061	0.071	Nucleotide Transformer
NABench FS-CORR-RIBOZYME: Fitness prediction on ribozyme assays, few-shot	NABench ribozyme assays (NABench split)	Spearman, unitless	0.09	0.135	Nucleotide Transformer
NABench RCV-CORR-APTAMER: Fitness prediction on aptamer assays, supervised, random cross validation	NABench aptamer assays (NABench split)	Spearman, unitless	0.462	0.36	Nucleotide Transformer v2
NABench RCV-CORR-ENHANCER: Fitness prediction on enhancer assays, supervised, random cross validation	NABench enhancer assays (NABench split)	Spearman, unitless	0.148	0.21	Nucleotide Transformer
NABench RCV-CORR-MRNA: Fitness prediction on mRNA assays, supervised, random cross validation	NABench mRNA assays (NABench split)	Spearman, unitless	0.57	0.56	Nucleotide Transformer v2
NABench RCV-CORR-PROMOTER: Fitness prediction on promoter assays, supervised, random cross validation	NABench promoter assays (NABench split)	Spearman, unitless	0.632	0.636	Tie
NABench RCV-CORR-RIBOZYME: Fitness prediction on ribozyme assays, supervised, random cross validation	NABench ribozyme assays (NABench split)	Spearman, unitless	0.464	0.429	Nucleotide Transformer v2
NABench RCV-CORR-TRNA: Fitness prediction on tRNA assays, supervised, random cross validation	NABench tRNA assays (NABench split)	Spearman, unitless	0.47	0.433	Nucleotide Transformer v2
NABench SELEX-FS: Overall fitness prediction on NABench SELEX assays, Few-shot Spearman ρ	NABench SELEX assays (NABench split)	Spearman, unitless	0.578	0.564	Nucleotide Transformer v2
NABench SELEX-RCV: Overall fitness prediction on NABench SELEX assays, Random cross validation Spearman ρ	NABench SELEX assays (NABench split)	Spearman, unitless	0.66	0.626	Nucleotide Transformer v2
NABench SELEX-ZS-AUC: Overall fitness prediction on NABench SELEX assays, Zero-shot AUC	NABench SELEX assays (NABench split)	AUROC, unitless	0.554	0.526	Nucleotide Transformer v2
NABench SELEX-ZS-CORR: Overall fitness prediction on NABench SELEX assays, Zero-shot Spearman ρ	NABench SELEX assays (NABench split)	Spearman, unitless	0.09	0.039	Nucleotide Transformer v2
NABench SELEX-ZS-MCC: Overall fitness prediction on NABench SELEX assays, Zero-shot MCC	NABench SELEX assays (NABench split)	MCC, unitless	0.047	0.027	Nucleotide Transformer v2
NABench SELEX-ZS-NDCG: Overall fitness prediction on NABench SELEX assays, Zero-shot NDCG	NABench SELEX assays (NABench split)	NDCG (cutoff not stated), unitless	0.36	0.195	Nucleotide Transformer v2
NABench ZS-AUC-APTAMER: Fitness prediction on aptamer assays, zero-shot	NABench aptamer assays (NABench split)	AUROC, unitless	0.56	0.524	Nucleotide Transformer v2
NABench ZS-AUC-ENHANCER: Fitness prediction on enhancer assays, zero-shot	NABench enhancer assays (NABench split)	AUROC, unitless	0.54	0.528	Nucleotide Transformer v2
NABench ZS-AUC-MRNA: Fitness prediction on mRNA assays, zero-shot	NABench mRNA assays (NABench split)	AUROC, unitless	0.526	0.521	Tie
NABench ZS-AUC-PROMOTER: Fitness prediction on promoter assays, zero-shot	NABench promoter assays (NABench split)	AUROC, unitless	0.676	0.573	Nucleotide Transformer v2
NABench ZS-AUC-RIBOZYME: Fitness prediction on ribozyme assays, zero-shot	NABench ribozyme assays (NABench split)	AUROC, unitless	0.554	0.537	Nucleotide Transformer v2
NABench ZS-CORR-APTAMER: Fitness prediction on aptamer assays, zero-shot	NABench aptamer assays (NABench split)	Spearman, unitless	0.092	0.081	Nucleotide Transformer v2
NABench ZS-CORR-ENHANCER: Fitness prediction on enhancer assays, zero-shot	NABench enhancer assays (NABench split)	Spearman, unitless	0.064	0.006	Nucleotide Transformer v2
NABench ZS-CORR-MRNA: Fitness prediction on mRNA assays, zero-shot	NABench mRNA assays (NABench split)	Spearman, unitless	0.068	0.074	Nucleotide Transformer
NABench ZS-CORR-PROMOTER: Fitness prediction on promoter assays, zero-shot	NABench promoter assays (NABench split)	Spearman, unitless	0.384	0.091	Nucleotide Transformer v2
NABench ZS-CORR-RIBOZYME: Fitness prediction on ribozyme assays, zero-shot	NABench ribozyme assays (NABench split)	Spearman, unitless	0.083	0.064	Nucleotide Transformer v2

Comparisons in Nucleotide Transformer (19)

Protocol	Dataset	Metric	Nucleotide Transformer v2	Nucleotide Transformer	Higher
NT benchmark Enhancer, fine-tuning (PEFT)	NT benchmark Enhancer	MCC (median over 10 cross-validation folds), unitless	0.524	0.504	Nucleotide Transformer v2
NT benchmark Enhancer types, fine-tuning (PEFT)	NT benchmark Enhancer types	MCC (median over 10 cross-validation folds), unitless	0.498	0.469	Nucleotide Transformer v2
NT benchmark H2AFZ, fine-tuning (PEFT)	NT benchmark H2AFZ	MCC (median over 10 cross-validation folds), unitless	0.493	0.478	Nucleotide Transformer v2
NT benchmark H3K27ac, fine-tuning (PEFT)	NT benchmark H3K27ac	MCC (median over 10 cross-validation folds), unitless	0.492	0.486	Nucleotide Transformer v2
NT benchmark H3K27me3, fine-tuning (PEFT)	NT benchmark H3K27me3	MCC (median over 10 cross-validation folds), unitless	0.613	0.603	Nucleotide Transformer v2
NT benchmark H3K36me3, fine-tuning (PEFT)	NT benchmark H3K36me3	MCC (median over 10 cross-validation folds), unitless	0.642	0.632	Nucleotide Transformer v2
NT benchmark H3K4me1, fine-tuning (PEFT)	NT benchmark H3K4me1	MCC (median over 10 cross-validation folds), unitless	0.497	0.491	Nucleotide Transformer v2
NT benchmark H3K4me2, fine-tuning (PEFT)	NT benchmark H3K4me2	MCC (median over 10 cross-validation folds), unitless	0.564	0.569	Tie
NT benchmark H3K4me3, fine-tuning (PEFT)	NT benchmark H3K4me3	MCC (median over 10 cross-validation folds), unitless	0.636	0.615	Nucleotide Transformer v2
NT benchmark H3K9ac, fine-tuning (PEFT)	NT benchmark H3K9ac	MCC (median over 10 cross-validation folds), unitless	0.539	0.529	Nucleotide Transformer v2
NT benchmark H3K9me3, fine-tuning (PEFT)	NT benchmark H3K9me3	MCC (median over 10 cross-validation folds), unitless	0.475	0.483	Nucleotide Transformer
NT benchmark H4K20me1, fine-tuning (PEFT)	NT benchmark H4K20me1	MCC (median over 10 cross-validation folds), unitless	0.658	0.659	Tie
NT benchmark normalised summed performance (Supplementary Table 9)	NT benchmark, all 18 tasks	MCC (mean over three task categories of median cross-validation MCC), unitless	0.762	0.738	Nucleotide Transformer v2
NT benchmark Promoter all, fine-tuning (PEFT)	NT benchmark Promoter all	MCC (median over 10 cross-validation folds), unitless	0.778	0.708	Nucleotide Transformer v2
NT benchmark Promoter non-TATA, fine-tuning (PEFT)	NT benchmark Promoter non-TATA	MCC (median over 10 cross-validation folds), unitless	0.774	0.758	Nucleotide Transformer v2
NT benchmark Promoter TATA, fine-tuning (PEFT)	NT benchmark Promoter TATA	MCC (median over 10 cross-validation folds), unitless	0.878	0.802	Nucleotide Transformer v2
NT benchmark Splice acceptor, fine-tuning (PEFT)	NT benchmark Splice acceptor	MCC (median over 10 cross-validation folds), unitless	0.944	0.952	Tie
NT benchmark Splice donor, fine-tuning (PEFT)	NT benchmark Splice donor	MCC (median over 10 cross-validation folds), unitless	0.96	0.963	Tie
NT benchmark Splice site all, fine-tuning (PEFT)	NT benchmark Splice site all	MCC (median over 10 cross-validation folds), unitless	0.967	0.956	Nucleotide Transformer v2

What disagrees

Other studies: rank Nucleotide Transformer v2 above Nucleotide Transformer in 33 of 40 comparisons
This study: ranks Nucleotide Transformer above Nucleotide Transformer v2 in 37 of 44 comparisons

Compared: Nucleotide Transformer v2, Nucleotide Transformer
Study that disagrees: GENEB leaderboard (leaderboard.json, 13 task-type aggregates)
Other studies: NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction, Nucleotide Transformer: building and evaluating robust foundation models for human genomics (Nature Methods), Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)
Rule: Ordering reversal; different-task
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (88)
Study	Tool	Protocol or benchmark	Metric	Value
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Chromatin Acc.')	0.41
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Coding/NC')	0.667
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'DNA Methyl.')	0.0575
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Enhancers')	0.314
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Histone Mod.')	0.258
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'lncRNA')	0.103
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; macro average: unweighted mean of the 13 category MCCs)	0.293
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; micro average: mean over all 100 tasks)	0.3
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Mouse Enh.')	0.202
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Promoters')	0.532
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Regulatory')	0.152
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Species Clf.')	0.512
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Splice Sites')	0.242
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'TF Binding')	0.201
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Virus/Phage')	0.161
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Chromatin Acc.')	0.169
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Coding/NC')	0.251
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'DNA Methyl.')	0.0179
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Enhancers')	0.171
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Histone Mod.')	0.0952
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'lncRNA')	0.0454
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; macro average: unweighted mean of the 13 category MCCs)	0.116
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; micro average: mean over all 100 tasks)	0.136
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Mouse Enh.')	0.0481
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Promoters')	0.309
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Regulatory')	0.0837
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Species Clf.')	0.192
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Splice Sites')	0.0768
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'TF Binding')	0.0361
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Virus/Phage')	0.0194
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Chromatin Acc.')	0.554
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Coding/NC')	0.864
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'DNA Methyl.')	0.356
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Enhancers')	0.487
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Histone Mod.')	0.498
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'lncRNA')	0.372
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; macro average: unweighted mean of the 13 category MCCs)	0.573
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; micro average: mean over all 100 tasks)	0.561
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Mouse Enh.')	0.547
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Promoters')	0.739
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Regulatory')	0.488
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Splice Sites')	0.652
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'TF Binding')	0.573
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-2.5B-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Virus/Phage')	0.564
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Chromatin Acc.')	0.386
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Coding/NC')	0.534
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'DNA Methyl.')	0.0411
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Enhancers')	0.288
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Histone Mod.')	0.261
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'lncRNA')	0.0528
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; macro average: unweighted mean of the 13 category MCCs)	0.247
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; micro average: mean over all 100 tasks)	0.263
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Mouse Enh.')	0.121
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Promoters')	0.482
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Regulatory')	0.178
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Species Clf.')	0.52
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Splice Sites')	0.0802
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'TF Binding')	0.125
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 10-shot, MCC aggregates	MCC (10-shot; category 'Virus/Phage')	0.144
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Chromatin Acc.')	0.176
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Coding/NC')	0.284
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'DNA Methyl.')	0.0159
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Enhancers')	0.159
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Histone Mod.')	0.0829
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'lncRNA')	0.00862
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; macro average: unweighted mean of the 13 category MCCs)	0.102
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; micro average: mean over all 100 tasks)	0.106
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Mouse Enh.')	0.0248
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Promoters')	0.222
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Regulatory')	0.0692
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Species Clf.')	0.21
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Splice Sites')	0.0478
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'TF Binding')	0.0202
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, 1-shot, MCC aggregates	MCC (1-shot; category 'Virus/Phage')	0.0062
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Chromatin Acc.')	0.652
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Coding/NC')	0.812
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'DNA Methyl.')	0.339
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Enhancers')	0.469
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Histone Mod.')	0.468
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'lncRNA')	0.337
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; macro average: unweighted mean of the 13 category MCCs)	0.537
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; micro average: mean over all 100 tasks)	0.524
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Mouse Enh.')	0.486
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Promoters')	0.722
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Regulatory')	0.457
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Splice Sites')	0.47
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'TF Binding')	0.537
GENEB leaderboard (leaderboard.json, 13 task-type aggregates)	NT-v2-250m-MS (GENEB leaderboard)	GENEB leaderboard, full training data, MCC aggregates	MCC (full training data; category 'Virus/Phage')	0.482

In the other studies (80 results)

In the other studies
Study	Tool	Protocol or benchmark	Metric	Value
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Enhancer, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.504 (± 0.009)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Enhancer types, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.469 (± 0.005)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H2AFZ, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.478 (± 0.012)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K27ac, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.486 (± 0.023)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K27me3, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.603 (± 0.009)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K36me3, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.632 (± 0.008)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K4me1, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.491 (± 0.015)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K4me3, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.615 (± 0.017)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K9ac, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.529 (± 0.012)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K9me3, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.483 (± 0.013)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Promoter all, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.708 (± 0.008)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Promoter non-TATA, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.758 (± 0.007)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Promoter TATA, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.802 (± 0.03)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Splice site all, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.956 (± 0.004)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-1000G (2.5B), parameter-efficient fine-tuning (NT benchmark)	NT benchmark normalised summed performance (Supplementary Table 9)	MCC (mean over three task categories of median cross-validation MCC)	0.738
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Enhancer, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.524 (± 0.013)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Enhancer types, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.498 (± 0.009)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H2AFZ, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.493 (± 0.01)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K27ac, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.492 (± 0.012)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K27me3, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.613 (± 0.01)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K36me3, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.642 (± 0.015)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K4me1, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.497 (± 0.009)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K4me3, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.636 (± 0.016)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K9ac, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.539 (± 0.013)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark H3K9me3, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.475 (± 0.014)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Promoter all, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.778 (± 0.012)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Promoter non-TATA, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.774 (± 0.015)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Promoter TATA, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.878 (± 0.021)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark Splice site all, fine-tuning (PEFT)	MCC (median over 10 cross-validation folds)	0.967 (± 0.004)
Nucleotide Transformer: Supplementary Information (Supplementary Tables 1-12)	NT-Multispecies-v2 (500M), parameter-efficient fine-tuning (NT benchmark)	NT benchmark normalised summed performance (Supplementary Table 9)	MCC (mean over three task categories of median cross-validation MCC)	0.762
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench FS-CORR-APTAMER: Fitness prediction on aptamer assays, few-shot	Spearman	0.166
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench FS-CORR-ENHANCER: Fitness prediction on enhancer assays, few-shot	Spearman	0.053
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench FS-CORR-MRNA: Fitness prediction on mRNA assays, few-shot	Spearman	0.256
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench FS-CORR-PROMOTER: Fitness prediction on promoter assays, few-shot	Spearman	0.071
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench FS-CORR-RIBOZYME: Fitness prediction on ribozyme assays, few-shot	Spearman	0.135
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench RCV-CORR-APTAMER: Fitness prediction on aptamer assays, supervised, random cross validation	Spearman	0.36
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench RCV-CORR-ENHANCER: Fitness prediction on enhancer assays, supervised, random cross validation	Spearman	0.21
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench RCV-CORR-MRNA: Fitness prediction on mRNA assays, supervised, random cross validation	Spearman	0.56
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench RCV-CORR-RIBOZYME: Fitness prediction on ribozyme assays, supervised, random cross validation	Spearman	0.429
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench RCV-CORR-TRNA: Fitness prediction on tRNA assays, supervised, random cross validation	Spearman	0.433
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench SELEX-FS: Overall fitness prediction on NABench SELEX assays, Few-shot Spearman ρ	Spearman	0.564
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench SELEX-RCV: Overall fitness prediction on NABench SELEX assays, Random cross validation Spearman ρ	Spearman	0.626
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench SELEX-ZS-AUC: Overall fitness prediction on NABench SELEX assays, Zero-shot AUC	AUROC	0.526
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench SELEX-ZS-CORR: Overall fitness prediction on NABench SELEX assays, Zero-shot Spearman ρ	Spearman	0.039
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench SELEX-ZS-MCC: Overall fitness prediction on NABench SELEX assays, Zero-shot MCC	MCC	0.027
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench SELEX-ZS-NDCG: Overall fitness prediction on NABench SELEX assays, Zero-shot NDCG	NDCG (cutoff not stated)	0.195
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench ZS-AUC-APTAMER: Fitness prediction on aptamer assays, zero-shot	AUROC	0.524
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench ZS-AUC-ENHANCER: Fitness prediction on enhancer assays, zero-shot	AUROC	0.528
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench ZS-AUC-PROMOTER: Fitness prediction on promoter assays, zero-shot	AUROC	0.573
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench ZS-AUC-RIBOZYME: Fitness prediction on ribozyme assays, zero-shot	AUROC	0.537
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench ZS-CORR-APTAMER: Fitness prediction on aptamer assays, zero-shot	Spearman	0.081
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench ZS-CORR-ENHANCER: Fitness prediction on enhancer assays, zero-shot	Spearman	0.006
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench ZS-CORR-MRNA: Fitness prediction on mRNA assays, zero-shot	Spearman	0.074
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench ZS-CORR-PROMOTER: Fitness prediction on promoter assays, zero-shot	Spearman	0.091
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-500m	NABench ZS-CORR-RIBOZYME: Fitness prediction on ribozyme assays, zero-shot	Spearman	0.064
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench FS-CORR-APTAMER: Fitness prediction on aptamer assays, few-shot	Spearman	0.23
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench FS-CORR-ENHANCER: Fitness prediction on enhancer assays, few-shot	Spearman	0.026
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench FS-CORR-MRNA: Fitness prediction on mRNA assays, few-shot	Spearman	0.214
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench FS-CORR-PROMOTER: Fitness prediction on promoter assays, few-shot	Spearman	0.061
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench FS-CORR-RIBOZYME: Fitness prediction on ribozyme assays, few-shot	Spearman	0.09
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench RCV-CORR-APTAMER: Fitness prediction on aptamer assays, supervised, random cross validation	Spearman	0.462
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench RCV-CORR-ENHANCER: Fitness prediction on enhancer assays, supervised, random cross validation	Spearman	0.148
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench RCV-CORR-MRNA: Fitness prediction on mRNA assays, supervised, random cross validation	Spearman	0.57
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench RCV-CORR-RIBOZYME: Fitness prediction on ribozyme assays, supervised, random cross validation	Spearman	0.464
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench RCV-CORR-TRNA: Fitness prediction on tRNA assays, supervised, random cross validation	Spearman	0.47
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench SELEX-FS: Overall fitness prediction on NABench SELEX assays, Few-shot Spearman ρ	Spearman	0.578
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench SELEX-RCV: Overall fitness prediction on NABench SELEX assays, Random cross validation Spearman ρ	Spearman	0.66
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench SELEX-ZS-AUC: Overall fitness prediction on NABench SELEX assays, Zero-shot AUC	AUROC	0.554
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench SELEX-ZS-CORR: Overall fitness prediction on NABench SELEX assays, Zero-shot Spearman ρ	Spearman	0.09
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench SELEX-ZS-MCC: Overall fitness prediction on NABench SELEX assays, Zero-shot MCC	MCC	0.047
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench SELEX-ZS-NDCG: Overall fitness prediction on NABench SELEX assays, Zero-shot NDCG	NDCG (cutoff not stated)	0.36
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench ZS-AUC-APTAMER: Fitness prediction on aptamer assays, zero-shot	AUROC	0.56
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench ZS-AUC-ENHANCER: Fitness prediction on enhancer assays, zero-shot	AUROC	0.54
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench ZS-AUC-PROMOTER: Fitness prediction on promoter assays, zero-shot	AUROC	0.676
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench ZS-AUC-RIBOZYME: Fitness prediction on ribozyme assays, zero-shot	AUROC	0.554
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench ZS-CORR-APTAMER: Fitness prediction on aptamer assays, zero-shot	Spearman	0.092
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench ZS-CORR-ENHANCER: Fitness prediction on enhancer assays, zero-shot	Spearman	0.064
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench ZS-CORR-MRNA: Fitness prediction on mRNA assays, zero-shot	Spearman	0.068
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench ZS-CORR-PROMOTER: Fitness prediction on promoter assays, zero-shot	Spearman	0.384
NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction	N.T.-v2-50m	NABench ZS-CORR-RIBOZYME: Fitness prediction on ribozyme assays, zero-shot	Spearman	0.083

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-7a45e33dabf92d71 · All contested results · How findings are made and followed up