rewirebio.iobenchmarks
Source

Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material) · Preprint

DOI: 10.1101/2025.02.11.637758Original source

What this source contributed

Records in this release that cite this source. A record can cite several sources.

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Model or configurationMetricValueDatasetEvaluation
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.292 unitlessTraitGym Complex traits: Adult heightBorzoi on TraitGym Complex traits, trait Adult height
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.615 unitlessTraitGym Complex traits: AlbuminBorzoi on TraitGym Complex traits, trait Albumin
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.455 unitlessTraitGym Complex traits: Albumin/Globulin ratioBorzoi on TraitGym Complex traits, trait Albumin/Globulin ratio
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.492 unitlessTraitGym Complex traits: Alkaline phosphataseBorzoi on TraitGym Complex traits, trait Alkaline phosphatase
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.459 unitlessTraitGym Complex traits: Balding Type 4Borzoi on TraitGym Complex traits, trait Balding Type 4
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.574 unitlessTraitGym Complex traits: Blood clot in the legBorzoi on TraitGym Complex traits, trait Blood clot in the leg
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.344 unitlessTraitGym Complex traits: Body mass indexBorzoi on TraitGym Complex traits, trait Body mass index
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.468 unitlessTraitGym Complex traits: CalciumBorzoi on TraitGym Complex traits, trait Calcium
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.457 unitlessTraitGym Complex traits: Estimated glomerular filtration rate (cystain C)Borzoi on TraitGym Complex traits, trait Estimated glomerular filtration rate (cystain C)
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.308 unitlessTraitGym Complex traits: Estimated heel bone mineral densityBorzoi on TraitGym Complex traits, trait Estimated heel bone mineral density
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.430 unitlessTraitGym Complex traits: FEV1/FVC ratioBorzoi on TraitGym Complex traits, trait FEV1/FVC ratio
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.515 unitlessTraitGym Complex traits: Gamma-glutamyl transferaseBorzoi on TraitGym Complex traits, trait Gamma-glutamyl transferase
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.475 unitlessTraitGym Complex traits: Hemoglobin A1cBorzoi on TraitGym Complex traits, trait Hemoglobin A1c
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.521 unitlessTraitGym Complex traits: High density lipoprotein cholesterolBorzoi on TraitGym Complex traits, trait High density lipoprotein cholesterol
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.434 unitlessTraitGym Complex traits: Mean corpuscular volumeBorzoi on TraitGym Complex traits, trait Mean corpuscular volume
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.561 unitlessTraitGym Complex traits: Monocyte countBorzoi on TraitGym Complex traits, trait Monocyte count
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.426 unitlessTraitGym Complex traits: Platelet countBorzoi on TraitGym Complex traits, trait Platelet count
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.457 unitlessTraitGym Complex traits: Pulse pressureBorzoi on TraitGym Complex traits, trait Pulse pressure
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.927 unitlessTraitGym Mendelian traits: Beta-thalassemiaBorzoi on TraitGym Mendelian traits, trait Beta-thalassemia
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.594 unitlessTraitGym Mendelian traits: Cartilage-hair hypoplasiaBorzoi on TraitGym Mendelian traits, trait Cartilage-hair hypoplasia
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.484 unitlessTraitGym Mendelian traits: Dwarfism (MOPD1)Borzoi on TraitGym Mendelian traits, trait Dwarfism (MOPD1)
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.914 unitlessTraitGym Mendelian traits: Hemophilia BBorzoi on TraitGym Mendelian traits, trait Hemophilia B
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.844 unitlessTraitGym Mendelian traits: Hypercholesterolemia-1Borzoi on TraitGym Mendelian traits, trait Hypercholesterolemia-1
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.315 unitlessTraitGym Mendelian traits: HyperferritinemiaBorzoi on TraitGym Mendelian traits, trait Hyperferritinemia
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.546 unitlessTraitGym Mendelian traits: Preaxial polydactyly IIBorzoi on TraitGym Mendelian traits, trait Preaxial polydactyly II
BorzoiAUPRC (best of zero-shot and linear probing; single trait; chromosome aggregation unreported)0.564 unitlessTraitGym Mendelian traits: Pulmonary fibrosisBorzoi on TraitGym Mendelian traits, trait Pulmonary fibrosis
Borzoi predicted tracks from abdominal adipose tissue (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.42 unitlessTraitGym Complex traits: High density lipoprotein cholesterolBorzoi tracks from abdominal adipose tissue on TraitGym Complex traits, trait High density lipoprotein cholesterol
Borzoi predicted tracks from adrenal gland (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.417 unitlessTraitGym Complex traits: High density lipoprotein cholesterolBorzoi tracks from adrenal gland on TraitGym Complex traits, trait High density lipoprotein cholesterol
Borzoi predicted tracks from adrenal gland (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.986 unitlessTraitGym Mendelian traits: Beta-thalassemiaBorzoi tracks from adrenal gland on TraitGym Mendelian traits, trait Beta-thalassemia
Borzoi predicted tracks from aorta (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.997 unitlessTraitGym Mendelian traits: Beta-thalassemiaBorzoi tracks from aorta on TraitGym Mendelian traits, trait Beta-thalassemia
Borzoi predicted tracks from CD4+ T cell (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.972 unitlessTraitGym Mendelian traits: Hypercholesterolemia-1Borzoi tracks from CD4+ T cell on TraitGym Mendelian traits, trait Hypercholesterolemia-1
Borzoi predicted tracks from CD8+ T cell (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.983 unitlessTraitGym Mendelian traits: Hypercholesterolemia-1Borzoi tracks from CD8+ T cell on TraitGym Mendelian traits, trait Hypercholesterolemia-1
Borzoi predicted tracks from CD14+ monocyte (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.559 unitlessTraitGym Complex traits: Monocyte countBorzoi tracks from CD14+ monocyte on TraitGym Complex traits, trait Monocyte count
Borzoi predicted tracks from erythroblast (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.423 unitlessTraitGym Complex traits: Hemoglobin A1cBorzoi tracks from erythroblast on TraitGym Complex traits, trait Hemoglobin A1c
Borzoi predicted tracks from hematopoietic progenitor (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.412 unitlessTraitGym Complex traits: Hemoglobin A1cBorzoi tracks from hematopoietic progenitor on TraitGym Complex traits, trait Hemoglobin A1c
Borzoi predicted tracks from hepatocyte (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)1.0 unitlessTraitGym Mendelian traits: Hemophilia BBorzoi tracks from hepatocyte on TraitGym Mendelian traits, trait Hemophilia B
Borzoi predicted tracks from HepG2 (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)1.0 unitlessTraitGym Mendelian traits: Hemophilia BBorzoi tracks from HepG2 on TraitGym Mendelian traits, trait Hemophilia B
Borzoi predicted tracks from HepG2 (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.975 unitlessTraitGym Mendelian traits: Hypercholesterolemia-1Borzoi tracks from HepG2 on TraitGym Mendelian traits, trait Hypercholesterolemia-1
Borzoi predicted tracks from HL-60 (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.559 unitlessTraitGym Complex traits: Monocyte countBorzoi tracks from HL-60 on TraitGym Complex traits, trait Monocyte count
Borzoi predicted tracks from K562 (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.449 unitlessTraitGym Complex traits: Hemoglobin A1cBorzoi tracks from K562 on TraitGym Complex traits, trait Hemoglobin A1c
Borzoi predicted tracks from liver (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.44 unitlessTraitGym Complex traits: High density lipoprotein cholesterolBorzoi tracks from liver on TraitGym Complex traits, trait High density lipoprotein cholesterol
Borzoi predicted tracks from liver (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)1.0 unitlessTraitGym Mendelian traits: Hemophilia BBorzoi tracks from liver on TraitGym Mendelian traits, trait Hemophilia B
Borzoi predicted tracks from neutrophil (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.559 unitlessTraitGym Complex traits: Monocyte countBorzoi tracks from neutrophil on TraitGym Complex traits, trait Monocyte count
Borzoi predicted tracks from stomach (TraitGym)AUPRC (highest AUPRC among the Borzoi tracks from one tissue; single trait; chromosome aggregation unreported)0.988 unitlessTraitGym Mendelian traits: Beta-thalassemiaBorzoi tracks from stomach on TraitGym Mendelian traits, trait Beta-thalassemia
CADD input feature (-) Freq100bpAUPRC (single CADD input feature as the score; chromosome aggregation unreported)0.509 unitlessTraitGym Mendelian traits, matched 9 controlsCADD input feature (-) Freq100bp on TraitGym Mendelian traits
CADD input feature (-) Freq10000bpAUPRC (single CADD input feature as the score; chromosome aggregation unreported)0.131 unitlessTraitGym Complex traits, matched 9 controlsCADD input feature (-) Freq10000bp on TraitGym Complex traits
CADD input feature EncodeDNase-maxAUPRC (single CADD input feature as the score; chromosome aggregation unreported)0.145 unitlessTraitGym Complex traits, matched 9 controlsCADD input feature EncodeDNase-max on TraitGym Complex traits
CADD input feature EncodetotalRNA-maxAUPRC (single CADD input feature as the score; chromosome aggregation unreported)0.348 unitlessTraitGym Mendelian traits, matched 9 controlsCADD input feature EncodetotalRNA-max on TraitGym Mendelian traits
CADD input feature ZooPriPhyloPAUPRC (single CADD input feature as the score; chromosome aggregation unreported)0.225 unitlessTraitGym Complex traits, matched 9 controlsCADD input feature ZooPriPhyloP on TraitGym Complex traits
CADD input feature ZooVerPhyloPAUPRC (single CADD input feature as the score; chromosome aggregation unreported)0.673 unitlessTraitGym Mendelian traits, matched 9 controlsCADD input feature ZooVerPhyloP on TraitGym Mendelian traits

Claims

Statements this source supports, each with where in the source it was checked.

  • access: GPN-Promoter: Access GPN-Promoter is available at https://huggingface.co/songlab/gpn-animal-promoter and its training dataset at https://huggingface.co/datasets/songlab/gpn-animal-promoter-dataset. ('Data, model and code availability' (page 14))
  • architecture_hyperparameters: GPN-Promoter: Architecture hyperparameters The architecture follows ByteNet: blocks alternating dilated convolutions and feed-forward layers. Table S6 gives window size 512, repeat weight 0.01, embedding dimension 1024, slim True, 64 convolutional blocks, convolutional kernel size 9 in the first block and 5 in the remaining blocks, a dilation schedule of 1, 2, 4, 8, 16, 32, 64, 128 repeating, the AdamW optimizer, weight decay 0.01, batch size 2048, 370 K steps, learning rate 1e-3 and a 1 K step learning rate warmup. Table 2 gives 152M parameters, a context size of 512 and 1,026 extracted features. Training took approximately two weeks on four NVIDIA A100 40GB GPUs. (Table 2 (page 6), Appendix B.2 and Table S6 (pages 22 to 23))
  • author_reported_models: TraitGym: Author reported models The benchmark's authors are also the authors of GPN-MSA (Benegas et al. 2025a) and introduce GPN-Promoter in this paper as 'This work', so the GPN-MSA and GPN-Promoter rows of the benchmark are author-reported. (Table 2 'Source' column (page 6) and Section 5 'Models' (page 6))
  • availability: TraitGym: Availability The benchmark is at https://huggingface.co/datasets/songlab/TraitGym with a leaderboard at https://huggingface.co/spaces/songlab/TraitGym-leaderboard, code to reproduce the analysis at https://github.com/songlab-cal/TraitGym, and a Google Colab notebook that evaluates a model in a few minutes. (Abstract (page 1) and 'Data, model and code availability' (page 14))
  • benchmarked_models: TraitGym: Benchmarked models Table 2 lists the benchmarked systems with their dependencies, parameter count, context size and number of extracted features: Enformer (246M parameters, context 196K, 5,138 extracted features); Sei (890M parameters, context 4K, 41 extracted features); Borzoi (186M parameters, context 524K, 7,617 extracted features); GPN-MSA (86M parameters, context 128, 770 extracted features); NT (2.5B parameters, context 6K, 2,562 extracted features); HyenaDNA (14M parameters, context 160K, 258 extracted features); Caduceus (8M parameters, context 131K, 514 extracted features); SpeciesLM (97M parameters, context 2K, 770 extracted features); AIDO.DNA (7B parameters, context 4K, 4,354 extracted features); Evo2 (40B parameters, context 8,192, 8,194 extracted features); GPN-Promoter (152M parameters, context 512, 1,026 extracted features); CADD (N/A parameters, context N/A, 114 extracted features); phastCons (N/A parameters, context N/A, N/A extracted features); phyloP (N/A parameters, context N/A, N/A extracted features). Evo2 was trained with a 1 M context size but uses a shorter context for variant effect prediction. The downloaded Hugging Face Hub checkpoints are GPN-MSA songlab/gpn-msa-sapiens; NT InstaDeepAI/nucleotide-transformer-2.5b-multi-species; HyenaDNA LongSafari/hyenadna-medium-160k-seqlen-hf; Caduceus kuleshov-group/caduceus-ps_seqlen-131k_d_model-256_n_layer-16; SpeciesLM johahi/specieslm-metazoa-upstream-k6; AIDO.DNA genbio-ai/AIDO.DNA-7B. Evo2 checkpoints Evo2_1b_base, Evo2_7b and Evo2_40b came from the official repository, Enformer and Borzoi from gReLU's Model Zoo, Sei scores from the Sei web server, and CADD v1.7 scores and annotations from the Washington GRCh38 whole_genome_SNVs_inclAnno file. phyloP-100v, phyloP-241m and phastCons-43p were taken from the UCSC hg38 phyloP100way, cactus241way phyloP and Zoonomia primate phastCons tracks. (Table 2 (page 6), Table S5 (page 23) and Appendix B.1 (pages 22 to 23))
  • cadd_training_overlap: TraitGym: Cadd training overlap CADD is the only benchmarked model trained on variants, and its training variants overlap around 1 per cent of the variants in the datasets. Table S10 gives the overlap counts: Mendelian traits positives 0 with CADD training positives and 0 with CADD training negatives; Mendelian traits negatives 18 and 19; complex traits positives 8 and 1; complex traits negatives 79 and 55. The paper states that CADD's positives and negatives are not defined from causal variant annotations and show no clear association with the positive or negative sets here, and that repeating the analysis with the overlapping variants removed left the results stable (Figure S3). (Section 6 'Mendelian traits' (page 8) and Table S10 (page 28))
  • complex_variant_definition: TraitGym: Complex variant definition Putative causal variants for 83 complex traits come from UK Biobank fine-mapping (Kanai et al. 2021, data version 3 December 2019): the posterior inclusion probability was averaged between FINEMAP and SuSiE, variants where the two disagreed by more than 5 per cent were excluded, positives are variants with averaged probability above 0.9 in any trait, and positives were further filtered to genome-wide significance p below 5e-8. Controls are variants with probability below 0.01 in all traits. (Section 4 'Complex traits' (pages 4 to 5) and Appendix A.2 (page 20))
  • consequence_and_trait_composition: TraitGym: Consequence and trait composition Distal non-exonic variants, the potential enhancers, are 10 per cent of the Mendelian traits dataset and 76 per cent of the complex traits dataset. Of the 1,140 putative causal variants for complex traits only 53 affect a disease trait, and 21 of the complex traits are diseases or disorders (Table S1); the dataset holds only 3 non-coding variants affecting the risk of Alzheimer's disease. 103 of the complex trait causal variants (9 per cent) overlap fine-mapped GTEx eQTL variants, with no such overlap for Mendelian trait variants; those eQTL-overlapping variants are enriched in exonic variants (Fisher's exact p = 8e-8) and, among non-exonic variants, have lower transcription start site distances (Mann-Whitney p = 4e-4). (Section 4 'Variant type annotation' (page 5), Section 6 'Results by trait' and 'eQTL colocalization' (page 12) with Table S1 (page 20))
  • control_matching: TraitGym: Control matching Nine control variants are sampled for each putatively causal variant. Chromosome and consequence are matched exactly; the controls minimise the Euclidean distance to the positive variant in a robust-scaled space of transcription start site distance, and, for complex traits, minor allele frequency and UK Biobank linkage disequilibrium score. The proportion of consequences, including candidate cis-regulatory element categories and their flanks, is kept identical between positives and controls. Where a chromosome and consequence had too few controls, positives were subsampled until there were at least nine controls per positive. The baseline AUPRC is therefore 0.1. (Section 4 'Matching positives and negatives' (page 5) and Appendix A.4 (page 22))
  • dataset_counts: TraitGym: Dataset counts Mendelian traits: 338 putatively causal variants, 3,380 variants in total, across 113 traits. Complex traits: 1,140 putatively causal variants, 11,400 variants in total, across 83 traits. (Table 1 (page 4))
  • evaluation_metric: TraitGym: Evaluation metric The metric is the area under the precision-recall curve, computed for each chromosome with a model trained on the remaining chromosomes and then averaged across chromosomes with weights by sample size. The baseline AUPRC is 0.1, the proportion of positives. Appendix B.4 states that the weighted average AUPRC is used to evaluate zero-shot scores as well, 'to allow easy comparison', even though it is not strictly necessary for them. Figure S7 repeats the zero-shot results with a global AUPRC and Figure S6 with AUROC. (Section 4 'Task definition' (page 5) and Appendix B.4 (page 24))
  • expanded_datasets: TraitGym: Expanded datasets Besides the nine-control datasets, the benchmark provides expanded datasets with millions of control variants per positive. For Mendelian traits no subsampling of negatives is needed; for complex traits the negatives are subsampled into 100 equal-size minor allele frequency bins to match the positive set's distribution. Figure 6 reports n=339 positives against 4.9M negatives for Mendelian traits and n=1.1K against 2.9M for complex traits, and only models with precomputed genome-wide zero-shot scores (CADD, GPN-MSA, phyloP and phastCons) were evaluated on them; the paper estimates that a slower model such as Caduceus would need about six months on one NVIDIA A100 80GB GPU. (Section 4 'Matching positives and negatives' (page 5), Section 6 'Results on expanded datasets' with Figure 6 (page 9), Appendix A.4 (page 22) and Appendix B.4 (page 24))
  • feature_and_score_definitions: TraitGym: Feature and score definitions Table 3 fixes what each class of model contributes. Functional-genomics-supervised models (Enformer, Borzoi) give l2 scores, the norm over spatial positions of the log2 fold change in predicted activity between the alternate and reference sequence, per track, plus their aggregation across tracks ('l2 of l2 scores', all tracks and within each assay type); the zero-shot score is the l2 of l2 over all tracks. Sei gives changes in sequence class scores, with the maximum absolute change as the zero-shot score. Self-supervised models give the log-likelihood ratio and its absolute value, and the embedding inner product for each hidden dimension, with the embedding inner product, l2 distance and cosine distance as zero-shot scores. The integrative model gives CADD input features and the CADD score, with the CADD score as the zero-shot score, and conservation scores are used directly. Predictions are always averaged over the forward and reverse strand for reverse-complement invariance. (Table 3 and Section 5 (pages 6 to 7) and Appendix B.3 (pages 23 to 24))
  • limitations: TraitGym official metrics source: Limitations ["Official benchmark repository by the TraitGym authors, who also developed GPN-MSA and GPN-Promoter; those rows are author_reported.", "Benchmark paper: Benegas, Eraslan and Song, bioRxiv 10.1101/2025.02.11.637758 v2 (2025-03-04, CC BY-NC-ND), outside the search window; its Figure 2 plots these metrics. Its Table S8 prints the same zero-shot AUPRCs rounded to three decimal places, on the same chromosome-weighted basis: Appendix B.4 says \"To allow easy comparison, we also use the weighted average AUPRC to evaluate zero-shot scores, even though it is not strictly necessary\", and the global-AUPRC version is reported separately in Supplementary Figure S7. All 100 cells match the values here and were reconciled rather than re-extracted, in data/omics/traitgym-20261116; the paper source is linked with companion_of.", "Hugging Face datasets are mutable; the revision hash pins this retrieval."] (Appendix B.4 (page 24); Section 'Additional metrics' (page 7); Table S8 (page 27))
  • linear_probing_protocol: TraitGym: Linear probing protocol Linear probing trains a ridge logistic regression pipeline in scikit-learn with imputation, standardisation and a grid search over the regularisation strength, using as many folds as there are chromosomes: each fold tests on one chromosome using a model trained on the rest, with the regularisation chosen by group K-fold cross-validation whose groups are the training chromosomes. For CADD, linear probing uses 113 CADD input features together with the single CADD output score; for the other models it uses only output features, that is predicted tracks, log-likelihood ratios or embedding similarity. (Section 5 'Models' (page 6), Figure 5 caption (page 8) and Appendix B.4 with Listing 1 (pages 24 to 25))
  • links:companion_of:traitgym-20261116-source-biorxiv-v2: TraitGym official metrics source: Companion of Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2) (Data, model and code availability (page 15); Table S8 (page 27); Appendix B.4 (page 24))
  • links:configuration_of:traitgym-20261116-model-gpn-promoter: GPN_final_absLLR.plus.score (TraitGym): Configuration of GPN-Promoter (Section 5 'Models' (page 8); Table 2 (page 8); Appendix B.2 and Table S6 (page 22); Table S8 (page 27); Data, model and code availability (page 15))
  • links:configuration_of:traitgym-20261116-model-gpn-promoter: GPN_final_Embeddings.minus.inner_product (TraitGym): Configuration of GPN-Promoter (Section 5 'Models' (page 8); Table 2 (page 8); Appendix B.2 and Table S6 (page 22); Table S8 (page 27); Data, model and code availability (page 15))
  • links:configuration_of:traitgym-20261116-model-gpn-promoter: GPN_final_Embeddings.plus.cosine_distance (TraitGym): Configuration of GPN-Promoter (Section 5 'Models' (page 8); Table 2 (page 8); Appendix B.2 and Table S6 (page 22); Table S8 (page 27); Data, model and code availability (page 15))
  • links:configuration_of:traitgym-20261116-model-gpn-promoter: GPN_final_Embeddings.plus.euclidean_distance (TraitGym): Configuration of GPN-Promoter (Section 5 'Models' (page 8); Table 2 (page 8); Appendix B.2 and Table S6 (page 22); Table S8 (page 27); Data, model and code availability (page 15))
  • links:configuration_of:traitgym-20261116-model-gpn-promoter: GPN_final_EuclideanDistance.plus.score (TraitGym): Configuration of GPN-Promoter (Section 5 'Models' (page 8); Table 2 (page 8); Appendix B.2 and Table S6 (page 22); Table S8 (page 27); Data, model and code availability (page 15))
  • links:configuration_of:traitgym-20261116-model-gpn-promoter: GPN_final_InnerProduct.minus.score (TraitGym): Configuration of GPN-Promoter (Section 5 'Models' (page 8); Table 2 (page 8); Appendix B.2 and Table S6 (page 22); Table S8 (page 27); Data, model and code availability (page 15))
  • links:configuration_of:traitgym-20261116-model-gpn-promoter: GPN_final_LLR.minus.score (TraitGym): Configuration of GPN-Promoter (Section 5 'Models' (page 8); Table 2 (page 8); Appendix B.2 and Table S6 (page 22); Table S8 (page 27); Data, model and code availability (page 15))
  • links:configuration_of:traitgym-20261116-model-gpn-promoter: GPN_final.LogisticRegression.chrom (TraitGym): Configuration of GPN-Promoter (Section 5 'Models' (page 8); Table 2 (page 8); Appendix B.2 and Table S6 (page 22); Table S8 (page 27); Data, model and code availability (page 15))
  • matching_feature_control: TraitGym: Matching feature control The features the controls were matched on carry almost no predictive power, which is the intent of the matching: 'Table S4 shows that the matched features have minimal predictive power, as intended.' The printed global AUPRCs are close to the 0.1 baseline. (Appendix A.4 with Table S4 (page 22))

Models, benchmarks and other records

Models (1)

Methods (10)

Configurations (15)

Benchmarks (1)

Protocols (8)

Dataset subsets (26)

Evaluations (132)

Sources (1)

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

29 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-21-adc15377fab2
Property and statementOriginal source and locationReview and provenance
Artifact SHA-256
80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa
Source metadata
Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

Original source ↗

No field-specific location recorded

Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Retrieved: 2026-11-16T01:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.artifact_sha256

Source artifact SHA-256: 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa

Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.

Inspected artifact

Artifact url
https://www.biorxiv.org/content/10.1101/2025.02.11.637758v2.full.pdf
Source metadata
Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

Original source ↗

No field-specific location recorded

Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Retrieved: 2026-11-16T01:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.artifact_url

Source artifact SHA-256: 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa

Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.

Inspected artifact

Doi
10.1101/2025.02.11.637758
Source metadata
Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

Original source ↗

No field-specific location recorded

Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Retrieved: 2026-11-16T01:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.doi

Source artifact SHA-256: 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa

Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.

Inspected artifact

Hash scope
SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.
Source metadata
Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

Original source ↗

No field-specific location recorded

Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Retrieved: 2026-11-16T01:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.hash_scope

Source artifact SHA-256: 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa

Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.

Inspected artifact

Licence
CC BY-NC-ND 4.0
Source metadata
Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

Original source ↗

No field-specific location recorded

Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Retrieved: 2026-11-16T01:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.licence

Source artifact SHA-256: 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa

Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.

Inspected artifact

Limitations
3 values
  • Not peer reviewed.
  • The benchmark's authors developed GPN-MSA and GPN-Promoter, so their rows are author_reported.
  • The headline comparisons are Figures 5 to 10 and Figures S1 to S11, which print no values, and no source data file is published for them.
Source metadata
Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

Original source ↗

No field-specific location recorded

Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Retrieved: 2026-11-16T01:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.limitations

Source artifact SHA-256: 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa

Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.

Inspected artifact

Media type
application/pdf
Source metadata
Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

Original source ↗

No field-specific location recorded

Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Retrieved: 2026-11-16T01:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.media_type

Source artifact SHA-256: 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa

Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.

Inspected artifact

Publication status
preprint
Source metadata
Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

Original source ↗

No field-specific location recorded

Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Retrieved: 2026-11-16T01:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.publication_status

Source artifact SHA-256: 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa

Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.

Inspected artifact

Retrieval note
Downloaded with curl from the v2 full-PDF URL. The supplementary material is part of the same PDF (pages 19 to 37); bioRxiv lists no separate supplementary file for this preprint.
Source metadata
Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

Original source ↗

No field-specific location recorded

Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Retrieved: 2026-11-16T01:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.retrieval_note

Source artifact SHA-256: 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa

Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.

Inspected artifact

Retrieved at
2026-11-16T01:09:36Z
Source metadata
Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2)

Original source ↗

No field-specific location recorded

Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Retrieved: 2026-11-16T01:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.retrieved_at

Source artifact SHA-256: 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa

Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.

Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-21-adc15377fab2 · Record review: source checked

Primary source retrieved and hashed for the 2026-11-16 GPN-family pass.

Release history

No supporting source is linked yet.

Read original source

Download this release (gzip)
Technical metadata and extraction receipts

Stable ID: traitgym-20261116-source-biorxiv-v2

areas
dna-genomes
contexts
research
artifact sha256
80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa
artifact url
https://www.biorxiv.org/content/10.1101/2025.02.11.637758v2.full.pdf
doi
10.1101/2025.02.11.637758
hash scope
SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer.
licence
CC BY-NC-ND 4.0
limitations
Not peer reviewed.; The benchmark's authors developed GPN-MSA and GPN-Promoter, so their rows are author_reported.; The headline comparisons are Figures 5 to 10 and Figures S1 to S11, which print no values, and no source data file is published for them.
media type
application/pdf
publication status
preprint
retrieval note
Downloaded with curl from the v2 full-PDF URL. The supplementary material is part of the same PDF (pages 19 to 37); bioRxiv lists no separate supplementary file for this preprint.
retrieved at
2026-11-16T01:09:36Z
review
artifact sha256: 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097; method: source-hash-verification; scope-review; method note: The pinned artifact was re-downloaded from the v2 full-PDF URL for this pass. Its PDF bytes hash to 961933ed2396ecb0a75737cc27f0655db07ca62be49654f9139ae95815d050f9, not to the 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa on the source record, and cannot: bioRxiv restamps the container on every request. The reviewer's download differs from the extractor's two in 41 and 44 bytes, every one of them a download-time stamp (the XMP ModifyDate and MetadataDate, the document ModDate and the trailer ID), with the same 37 pages, the same CreationDate of 2025-03-04 and the same pdftotext -layout text layer. That text layer hashes to 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097, which is the hash the source record's hash_scope names as the stable check and the hash recorded here; every value in this batch was read from it.; note: Publication status was re-checked independently on 2026-11-16 and the work is still a preprint only. The bioRxiv API lists versions 1 and 2 with published "NA" and its /pubs endpoint reports no published version; the v2 article page carries no "Now published in" banner; Crossref holds only the posted-content record with an empty relation object and a bibliographic search returns no journal article; PubMed has 39990426, the preprint, as the only match for TraitGym, and the eight Benegas G records for 2025 to 2027 contain no journal version of this work (the one Nature 2026 paper, 42717086, is GPN-Star); Europe PMC holds PPR978187 only; OpenAlex types the work as a preprint with bioRxiv, PubMed and PMC locations and no journal; Semantic Scholar gives the venue as bioRxiv. Searches were by author and by topic as well as by title, because a journal version can appear under a changed title. The pinned artifact was re-downloaded from the v2 full-PDF URL for this pass. Its PDF bytes hash to 961933ed2396ecb0a75737cc27f0655db07ca62be49654f9139ae95815d050f9, not to the 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa on the source record, and cannot: bioRxiv restamps the container on every request. The reviewer's download differs from the extractor's two in 41 and 44 bytes, every one of them a download-time stamp (the XMP ModifyDate and MetadataDate, the document ModDate and the trailer ID), with the same 37 pages, the same CreationDate of 2025-03-04 and the same pdftotext -layout text layer. That text layer hashes to 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097, which is the hash the source record's hash_scope names as the stable check and the hash recorded here; every value in this batch was read from it.; retrieval url: https://www.biorxiv.org/content/10.1101/2025.02.11.637758v2.full.pdf; reviewed at: 2026-11-16T02:10:00Z; reviewer: claude; reviewer note: Claude review agent (independent of the extractor); no human review claimed. Every value was read against the pinned artifact; nothing was reproduced and no model was run.
url
https://www.biorxiv.org/content/10.1101/2025.02.11.637758v2
version
bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
version note
The bioRxiv API lists two versions, v1 of 2025-02-12 and v2 of 2025-03-04, and no published version. PubMed (39990426), Europe PMC (PPR978187) and Crossref hold the preprint only, and a Crossref title search and a PubMed author search for Benegas G on 2026-11-16 returned no journal article of this work. The preprint is the version of record.
Related records

Suggest a correction