rewirebio.iobenchmarks

Use caseDNA and genomesResearch only

Select regulatory variants and genes for functional follow-up

Which variants, regulatory elements and genes should I perturb to explain a disease-associated locus?

You bring
  • Fine-mapped variants with genome build, ancestry and linkage-disequilibrium context
  • Candidate regulatory elements, genes and a relevant cell type
  • Sequence and available chromatin, expression or contact measurements
  • The feasible perturbation assay, readout and follow-up budget
You want
A prioritised set of variant–element–gene hypotheses, with the experiments needed to test each link and effect direction where supported.
Evidence in this release
11 comparisons shown · 52 tools · 189 configurations
Kept up to date
Evidence last checked 14 October 2026 · re-checked monthly, next due 14 November 2026
Executive Takeaway

For "Select regulatory variants and genes for functional follow-up", on TraitGym official metrics, Mendelian traits, zero-shot (AUPRC, a proxy protocol), CADD.plus.RawScore scored 0.7122, reported by TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym).

Best reported valueCADD.plus.RawScoreTraitGym official metrics: AUPRC by chromosome, weighted…
AUPRC0.7122TraitGym official metrics, Mendelian traits, zero-shot
Baseline on this protocolNone scoredNo margin is computed
Ranked tools6350 from independent studies

63 tools have a value on this protocol; 50 of them come from studies whose authors did not develop the tool. No baseline method is scored on this protocol. Reviewers judged this protocol a proxy for the use case.

Ranking

Ranked on TraitGym official metrics, Mendelian traits, zero-shot (AUPRC, higher is better). Proxy protocol Reviewers judged this protocol a proxy for the use case, not a direct test of it. Only results on this protocol are ranked; other tools are listed below the table.

14 tools in 63 rows, from 1 source. 50 of 63 results (79%) come from an independent study; 13 are reported by the tool's developers.

Rank▲ModelAUPRC (higher is better)EvidenceCoverageReported byLicense
#10.7122Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#20.6960Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#30.6945Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Code: MIT (songlab-cal/gpn)…
#40.6548Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#50.6542Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Code: MIT (songlab-cal/gpn)…
#60.5565Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#70.5275Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#80.5219Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#90.4628Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#100.4356Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#110.4217Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#120.4182Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#130.3849Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#140.3788Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#150.3506Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#160.3470Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#170.3450Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#180.3399Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#190.3271Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#200.3245Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#210.3011Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Code: MIT (songlab-cal/gpn)…
#220.3011Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Code: MIT (songlab-cal/gpn)…
#230.2911Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#240.2634Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#250.2544Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#260.2080Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Code: MIT (songlab-cal/gpn)…
#270.2069Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Code: MIT (songlab-cal/gpn)…
#280.2015Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#290.1878Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#300.1876Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Code: CC-BY-NC-SA-4.0
#310.1856Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Code: CC-BY-NC-SA-4.0
#320.1849Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Code: CC-BY-NC-SA-4.0
#330.1849Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Code: CC-BY-NC-SA-4.0
#340.1734Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#350.1693Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#360.1692Source-checked Developer-reported1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#370.1649Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#380.1648Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#390.1648Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#400.1608Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#410.1605Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#420.1585Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#430.1532Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#440.1477Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#450.1461Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#460.1396Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#470.1349Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#480.1347Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#490.1346Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#500.1311Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#510.1311Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#520.1282Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#530.1200Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Code: CC-BY-NC-SA-4.0
#540.1181Source-checked Independent study2/11TraitGym official metrics: AUPRC by chromosome…Apache-2.0 is declared in…
#550.1169Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#560.1164Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#570.1152Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#580.1148Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#590.1077Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#600.1065Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#610.0981Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Code: CC-BY-NC-SA-4.0
#620.0955Source-checked Independent study1/11TraitGym official metrics: AUPRC by chromosome…Not recorded
#630.0878Source-checked Independent study3/11TraitGym official metrics: AUPRC by chromosome…Not recorded
80 tools are evaluated elsewhere in this use case but not on this protocol, so not ranked

Baseline

Distance to TSS and Activity-by-Contact. Ranking genes by distance to the TSS and the Activity-by-Contact score (element activity times 3D contact) are the established non-learned rules for linking elements to genes. Learned linkers such as ENCODE-rE2G, a supervised model that extends ABC, are judged by how much they improve on them.

Machine-learning methods are compared with the baseline on the same data in 2 tables: MPRabc K562 CRISPRi enhancer–gene evaluation, CRISPR enhancer-gene benchmark, weighted metrics (Gschwind et al. 2026). Baseline rows are marked in each table.

Tools compared

Comparisons

Unified model comparison for this use case. Compare tools against protocol baselines in a single view.

Rows marked Contested rest on results that other studies disagree with. The results are still shown. See the disagreements that involve this use case for what disagrees and what we are doing about it.

TraitGym official metrics

Proxy evidenceIndependent study: Reported in a study whose authors did not develop the evaluated tool.Developer-reported: Reported by the developers of the evaluated tool.

AUPRC (higher is better), with the other metrics below each value. Showing the top 10 of 18 rows. The best value in each column is highlighted.

ToolMendelian traits, zero-shotMendelian traits, linear-probingComplex traits, zero-shotComplex traits, linear-probing
CADD.plus.RawScore2 sweepsmachine learning
0.7122 (best in column)Not reported0.2499 (best in column)Not reported
0.6960Not reported0.2274Not reported
GPN-MSA_LLR.minus.score7 sweepsfoundation model
0.6945Not reported0.2123Not reported
evo2_40b_LLR.minus.score18 sweepsfoundation model
0.5565Not reported0.1372Not reported
0.5275Not reported0.2371Not reported
Borzoi_L2_L2.plus.all2 sweepsmachine learning
0.4356Not reported0.2357Not reported
GPN_final_LLR.minus.score8 sweepsfoundation model
0.4217Not reported0.1123Not reported
Enformer_L2_L2.plus.all2 sweepsmachine learning
0.4182Not reported0.2451Not reported
0.3271Not reported0.1348Not reported
Sei.plus.seqclass_max_absdiff2 sweepsmachine learning
0.2911Not reported0.1928Not reported
0.1876Not reported0.1179Not reported
0.1734Not reported0.1092Not reported
0.1648Not reported0.1177Not reported
0.1349Not reported0.1147Not reported
Not reported0.757Not reported0.351
Not reported0.846Not reported0.297
Not reported0.740Not reported0.362 (best in column)
Not reported0.722Not reported0.349
Show all 18 rowsShow the top 10 rows only
Caveats and method

What each column measures:

  • Mendelian traits, zero-shot: AUPRC (per-chromosome weighted average, with bootstrap SE) of zero-shot scores from gLMs, sequence-to-function models, CADD and conservation for Mendelian traits causal versus matched noncoding variants
  • Mendelian traits, linear-probing: AUPRC (per-chromosome weighted average, with bootstrap SE) of linear-probing scores from gLMs, sequence-to-function models, CADD and conservation for Mendelian traits causal versus matched noncoding variants
  • Complex traits, zero-shot: AUPRC (per-chromosome weighted average, with bootstrap SE) of zero-shot scores from gLMs, sequence-to-function models, CADD and conservation for Complex traits causal versus matched noncoding variants
  • Complex traits, linear-probing: AUPRC (per-chromosome weighted average, with bootstrap SE) of linear-probing scores from gLMs, sequence-to-function models, CADD and conservation for Complex traits causal versus matched noncoding variants
  • Benchmark authors developed GPN-MSA and GPN-Promoter (author_reported).
  • Mendelian positives are curated regulatory pathogenic variants, enriched for conserved promoter and UTR sites, which favours conservation scores and CADD.
  • Labels are not experimental allele effects.
  • Official metrics files on a mutable Hugging Face dataset, pinned by revision.
  • Linear-probing scores are fitted on TraitGym labels with leave-one-chromosome-out training, so they are not comparable with zero-shot scores.
  • Complex-trait positives come from UK Biobank fine-mapping (high posterior inclusion probability, mostly European ancestry).

Protocols: TraitGym Mendelian traits, zero-shot, AUPRC by chromosome weighted average, TraitGym Mendelian traits, linear-probing, AUPRC by chromosome weighted average, TraitGym Complex traits, zero-shot, AUPRC by chromosome weighted average, TraitGym Complex traits, linear-probing, AUPRC by chromosome weighted average

Primary benchmark evaluation shown (TraitGym official metrics). 10 additional study protocols (MPRabc K562 CRISPRi enhancer–gene evaluation, Feng eQTL discrimination, Feng ipaQTL discrimination and others) are cited in individual tool drawer rows and under Question, sources and review.

Not covered yet

  • Automated source review only; independent human scientific review remains outstanding.
  • No allele-specific endogenous-edit benchmark or equal-budget prospective shortlist benefit established.
  • Training/test independence unresolved: paper explicitly says benchmarking pipeline does not perform cross-validation.
  • No locus-to-gene comparison with printed per-method precision and recall was found. CALDERA (Schipper et al. 2026, PLoS Genet) prints only Brier scores for CALDERA, FLAMES, L2G and cS2G (Table 1); its AUPRC comparison is figure-only.
  • No benchmark of variant-effect predictors against endogenous allele edits (base, prime or saturation editing of noncoding sites) with printed per-method values was found; the linked variant comparisons use reporter assays.
  • The linked element-gene linking comparisons are by the developers of ENCODE-rE2G and ABC and their co-authors; the astrocyte CRISPRi benchmark (Green et al. 2026, Nat Neurosci), an independent test of ABC and ENCODE-rE2G, reports per-model values only in figures, prose and bootstrap source data.
  • No linked comparison scores long-context sequence models (Enformer, Borzoi) on reporter allelic effects with a documented input setup; Manzo et al. 2025 describe it two ways, so those rows are excluded.
  • Gschwind et al. Supplementary Tables 5 and 7 (eQTL and GWAS enrichment benchmarks) and the per-predictor K562 sheets of Table 3 are not extracted.
  • Manzo et al. causal-SNP top-k results within LD blocks are figure-only.

Contribute evidence or propose a correction

How to cite

Select regulatory variants and genes for functional follow-up. Rewire biological model benchmark database, release 2026-11-21-adc15377fab2. https://benchmarks.rewirebio.io/use-cases/regulatory-variant-gene-follow-up/

Release 2026-11-21-adc15377fab2. Rankings and values change between releases, so keep the release id in the citation. Cite the original publication as well for any value you reuse: each result links its source.

The compilation and annotations made by Rewire in this release are offered under CC BY 4.0; values and text taken from cited sources stay under the terms of those sources and are not covered by this licence. Data licence

Question, sources and review

Who this is for, setting and exclusions

Choose allele edits, regulatory-element perturbations and gene readouts that can distinguish competing explanations for a locus.

Who this is for
  • Functional geneticists
  • Researchers interpreting disease-associated loci
Biological setting
Research follow-up of defined disease-associated loci in a specified cell system. Allele-effect prediction, element–gene linking and experimental shortlist selection are separate comparisons.
Research setting
Research only. Molecular effects and regulatory links do not establish clinical variant classification or diagnostic validity.

Outside this use case

  • Reporter activity does not by itself establish endogenous regulation.
  • Perturbing an entire regulatory element is different from editing one allele.
  • A regulatory link alone does not establish a gene’s causal role in disease.
Evidence collection plan
Evidence collection planCollecting evidence

Already on record: 11 comparisons shown · 52 tools, in the evaluated evidence above. The status above describes only the specific comparison in this plan, which remains open; it does not mean no evidence has been collected. The plan defines a comparison to investigate; it does not establish model performance or suitability.

Comparison question

Does a method select more experimentally supported regulatory hypotheses than compatible distance or linking baselines at the same follow-up budget?

Baselines, outcomes and validation requirements

Baselines to include

  • Distance-based variant/gene or enhancer/gene selection where applicable
  • ABC, ENCODE-rE2G or scE2G for compatible element–gene linking tasks
  • Sequence predictors compared only on the allele-effect outputs they produce

Outcomes to measure

  • Allele-dependent regulatory effects
  • Endogenous element–gene and allele–gene effects, measured separately
  • Confirmed follow-up hits per assay budget

Validation requirements

  • Define the allele-effect, linking or selection endpoint before choosing a comparison.
  • Hold out loci and studies; audit linkage-disequilibrium and pretraining overlap.
  • Preserve cell context, genome build, assay and perturbation type.
  • Keep reporter activity, whole-element perturbation, single-base effects and disease causality distinct.

Next collection task

Close the remaining endpoint, independence and comparator gaps documented in the 30 September 2026 audit and linked article issue; obtain qualified scientific review.

Related articles
Sources, review and downloads

Automated source review · 2026-09-30 · Codex source curation, independent worker cross-review and root integration review

Source-backed literature curation with independent automated transcription and scope checks. No new model execution, independent experimental replication, qualified human scientific review or clinical validation.

Release 2026-11-21-adc15377fab2 · question use-case-regulatory-variant-gene-follow-up

Use-case input digest bfbb7ab9303becad1e3cb395568c26f41e72ddb67cd1a89a06d9eaa0f76bd95f

Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)