rewirebio.iobenchmarks

Use caseCells and tissuesResearch only

Assess models for genetic perturbation experiments

Which prediction methods and controls should I test before using expression predictions to plan genetic perturbation experiments?

What the evidence shows

The five sources whose comparisons are shown on this page score how well models predict expression changes after genetic perturbations they were not trained on.

Each uses its own split, gene set and metric, so their values are not pooled. Ahlmann-Eltze et al. 2025, who did not develop the deep models and ran them all themselves, scored held-out two-gene CRISPRa perturbations in K562 cells (Norman), training on every single-gene perturbation and half of the pairs. A baseline that adds the two single-gene effects had a mean L2 distance of 2.766 and a Pearson delta of 0.892 over the 1,000 most highly expressed genes, against 3.478 and 0.773 for Geneformer with a linear decoder, 4.096 and 0.631 for GEARS, 4.279 and 0.610 for both UCE and scBERT, 4.616 and 0.527 for scGPT, 4.722 and 0.537 for scFoundation, and 12.205 and 0.146 for CPA; predicting no change gave an L2 distance of 5.616.

Rewire computed these means from the authors' Source Data, no spread is stated, and the UCE and scBERT runs used ground-truth expression of the perturbed genes, a possible test leakage. In PerturBench v4, a preprint whose authors trained every model and built the Linear, Latent Additive and Decoder baselines, the cosine similarity of predicted log fold change on held-out Norman pairs ranged from 0.41 for Biolord and 0.44 for GEARS to 0.79 for the Latent Additive baseline, with CPA at 0.76 and the Linear baseline at 0.60, each a mean over five training seeds. When effects were transferred to melanoma co-culture (Frangieh21) every model scored between 0.01 and 0.22 and every printed DEG recall was 0.000; on held-out cell line and cytokine states (Jiang24) scores ranged from 0.17 for Linear to 0.64 for the Decoder baseline, and the paper describes the held-out states in two different ways.

In Csendes et al. 2025, whose authors are employees of Turbine Ltd. and built the baselines themselves, predicting the mean training response gave a higher Pearson delta than scGPT and scFoundation, as re-run by the same authors, on all four Perturb-seq datasets: 0.711, 0.557, 0.373 and 0.628 for Adamson, Norman, Replogle K562 and Replogle RPE1, against 0.641, 0.554, 0.327 and 0.596 for scGPT and 0.552, 0.459, 0.269 and 0.471 for scFoundation. Each dataset there has one split and one run per model with no spread printed, so small gaps such as 0.557 against 0.554 cannot be tested. The GEARS developers report a mean squared error of 0.216 for GEARS on Norman, against 0.354 for CPA and 0.551 for a no-change control, with spreads of unstated type.

PertEval-scFM, whose authors trained GEARS from scratch, reports an area under its distribution-shift curve of 0.815 for GEARS against 4.484 for an MLP and 4.612 for a mean baseline, lower being better. All of these scores measure agreement with observed expression in held-out perturbations; they do not show which experiments will succeed.

You bring
  • Single-cell expression measurements from perturbed and unperturbed cells, with perturbation and cell-type labels
  • A study design with multiple measured perturbations and cells per perturbation; combinations require combination examples during GEARS training
You want
Predicted post-perturbation expression and changes relative to unperturbed controls, assessed against measured responses.
Evidence in this release
6 comparisons shown · 8 held for review · 29 tools · 42 configurations
Kept up to date
Evidence last checked 9 October 2026 · re-checked monthly, next due 9 November 2026
Executive Takeaway

For "Assess models for genetic perturbation experiments", on Foundation models against simple controls on unseen perturbations (Csendes et al. 2025), Adamson (K562 CRISPRi) (Pearson delta, a proxy protocol), KNN_scGPT scored 0.753, reported by Csendes et al. 2025, Supplementary Material 4 (Supplementary Tables 1-3).

Best reported valueKNN_scGPTCsendes et al. 2025, Supplementary Material 4…
Pearson delta0.753Foundation models against simple controls on unseen perturbations (Csendes et al. 2025), Adamson (K562 CRISPRi)
Baseline on this protocolmean0.711
Ranked tools152 from independent studies

15 tools have a value on this protocol; 2 of them come from studies whose authors did not develop the tool. The use case's baseline, mean, scored 0.711 on the same protocol. Reviewers judged this protocol a proxy for the use case.

Ranking

Ranked on Foundation models against simple controls on unseen perturbations (Csendes et al. 2025), Adamson (K562 CRISPRi) (Pearson delta, higher is better). Proxy protocol Reviewers judged this protocol a proxy for the use case, not a direct test of it. Only results on this protocol are ranked; other tools are listed below the table.

6 tools in 15 rows, from 1 source. 2 of 15 results (13%) come from an independent study; 13 are reported by the tool's developers. The top-ranked result is reported by the tool's developers.

Rank▲ModelPearson delta (higher is better)vs meanEvidenceCoverageReported byLicense
#10.753+5.8%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#20.739+3.9%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#30.727+2.3%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#40.720+1.2%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#50.718+1.0%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#60.713+0.2%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#70.711+0.0%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#8
mean[-] Baseline
0.711Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#90.711-0.0%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#100.706-0.7%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#110.694-2.4%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#120.642-9.7%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#130.641-9.9%Source-checked Independent study2/6Csendes et al. 2025, Supplementary Material 4…Code: MIT for the…
#140.629-11.6%Source-checked Developer-reported1/6Csendes et al. 2025, Supplementary Material 4…Not recorded
#150.552-22.4%Source-checked Independent study2/6Csendes et al. 2025, Supplementary Material 4…Separate Model License; the…
23 tools are evaluated elsewhere in this use case but not on this protocol, so not ranked

Baseline

Training mean, no change, additive and linear models. There is no pre-machine-learning method for predicting expression responses; the established comparators are simple statistical controls: the training mean, no change, the sum of single-perturbation effects and a linear model. Benchmarks report that foundation models often do not beat them.

Machine-learning methods are compared with the baseline on the same data in 2 tables: Deep learning and foundation models against simple baselines (Ahlmann-Eltze et al. 2025), Foundation models against simple controls on unseen perturbations (Csendes et al. 2025). Baseline rows are marked in each table.

Tools compared

Comparisons

Unified model comparison for this use case. Compare tools against protocol baselines in a single view.

Rows marked Contested rest on results that other studies disagree with. The results are still shown. See the disagreements that involve this use case for what disagrees and what we are doing about it.

Foundation models against simple controls on unseen perturbations (Csendes et al. 2025)

Proxy evidenceDeveloper-reported: Reported by the developers of the evaluated tool.Independent study: Reported in a study whose authors did not develop the evaluated tool.

Pearson delta (higher is better), with the other metrics below each value. The best value in each column is highlighted.

ToolAdamson (K562 CRISPRi)Norman (K562 CRISPRa, singles and pairs)Replogle K562 (CRISPRi)Replogle RPE1 (CRISPRi)
KNN_scGPT4 sweepsconventional
0.753 (best in column)0.6030.3740.5805
RF_scElmo4 sweepsconventional
0.7060.663 (best in column)0.4710.6506 (best in column)
EN_scGPT4 sweepsconventional
0.7180.6140.3760.6288
meanBaselineconventional
0.7110.5570.3730.6284
0.6410.5540.3270.5964
0.5520.4590.2690.4709
Caveats and method

What each column measures:

  • Adamson (K562 CRISPRi): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Adamson (K562 CRISPRi)
  • Norman (K562 CRISPRa, singles and pairs): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Norman (K562 CRISPRa, singles and pairs)
  • Replogle K562 (CRISPRi): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Replogle K562 (CRISPRi)
  • Replogle RPE1 (CRISPRi): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Replogle RPE1 (CRISPRi)
  • Datasets have low perturbation-specific variance (Results, second section), so differences between models are small.
  • Top-20 DE metrics include the CRISPR target gene unless stated, which favours scGPT (claim).
  • All authors are full-time employees of Turbine Ltd., and one is a founder (Competing interests). They built the Train Mean and regressor baselines, configured and ran the scGPT and scFoundation fine-tuning themselves, and argue that the baselines win; the foundation models were not run by their developers, so their configuration may not be the best available.
  • One GEARS perturbation-exclusive split per dataset, within one cell line, with one fine-tuning run per foundation model (best epoch on validation); no seeds, repeats or spread are printed, so small differences between models are not testable.
  • The Wilcoxon top-20 DE metric is blank for scGPT and scFoundation in Supplementary Table 2.
  • Results paragraphs 7 and 8 cite Supplementary Table 1 for values that are in Supplementary Table 2.
  • All perturbations are single-gene CRISPRi in one cell line, so the use case's combination-training and cross-cell-type exclusions are not engaged.
  • 79 of the 116 test perturbations are two-gene combinations (9 with neither gene seen in training, 52 with one, 18 with both) and 37 are unseen single genes (Supplementary Table 1); every value pools them. The per-subgroup values (Supplementary Table 3) are not extracted because their metric is not labelled.
  • The make-up of the 138 training perturbations is not printed, so whether training included combination examples, which the use case requires for GEARS combination prediction, cannot be confirmed from this source. scFoundation runs through GEARS, so the use case's GEARS combination exclusion applies to it directly.
  • The regressors represent a combination as the sum of the two genes' embeddings (Methods 'Baseline models').

Protocols: Adamson et al. 2016 Perturb-seq, K562 CRISPRi single perturbations: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2), Norman et al. 2019 Perturb-seq, K562 CRISPRa single and double perturbations: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2), Replogle et al. 2022 genome-wide Perturb-seq subset, K562 CRISPRi: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2), Replogle et al. 2022 genome-wide Perturb-seq subset, RPE1 CRISPRi: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2)

Primary benchmark evaluation shown (Foundation models against simple controls on unseen perturbations (Csendes et al. 2025)). 5 additional study protocols (GEARS Norman2019 CPA-control comparison MSE: Norman2019 perturbation-response MSE, GEARS Norman2019 CPA-control comparison Pearson DE: Norman2019 perturbation-response Pearson DE, PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits and others) are cited in individual tool drawer rows and under Question, sources and review.

Held for review

Recorded but not shown as evidence until the reason below is resolved.

Harmonised single-cell and spatial foundation model benchmark (Chen et al. 2026) (Adamson K562 Perturb-seq)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-chen2026-eval-perturbation-adamson-cellplm, lit-cells-networks-20261011-chen2026-eval-perturbation-adamson-genept-w, lit-cells-networks-20261011-chen2026-eval-perturbation-adamson-scelmo, lit-cells-networks-20261011-chen2026-eval-perturbation-adamson-scgpt-spatial); no evaluation on the protocol is eligible evidence.

Harmonised single-cell and spatial foundation model benchmark (Chen et al. 2026) (Norman K562 Perturb-seq)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-chen2026-eval-perturbation-norman-cellplm, lit-cells-networks-20261011-chen2026-eval-perturbation-norman-genept-w, lit-cells-networks-20261011-chen2026-eval-perturbation-norman-scelmo, lit-cells-networks-20261011-chen2026-eval-perturbation-norman-scgpt-spatial); no evaluation on the protocol is eligible evidence.

Harmonised single-cell and spatial foundation model benchmark (Chen et al. 2026) (Replogle K562 Perturb-seq)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-chen2026-eval-perturbation-replogle-cellplm, lit-cells-networks-20261011-chen2026-eval-perturbation-replogle-genept-w, lit-cells-networks-20261011-chen2026-eval-perturbation-replogle-scelmo, lit-cells-networks-20261011-chen2026-eval-perturbation-replogle-scgpt-spatial); no evaluation on the protocol is eligible evidence.

DE classification benchmark (Sun et al. 2026) (Held-out cell line)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-sun2026de-eval-cross-average-1-nn, lit-cells-networks-20261011-sun2026de-eval-cross-average-10-nn, lit-cells-networks-20261011-sun2026de-eval-cross-average-logistic-regression, lit-cells-networks-20261011-sun2026de-eval-cross-average-majority-vote, lit-cells-networks-20261011-sun2026de-eval-cross-average-scvi-mean); no evaluation on the protocol is eligible evidence.

DE classification benchmark (Sun et al. 2026) (Unseen perturbations within a cell line)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-sun2026de-eval-within-average-1-nn, lit-cells-networks-20261011-sun2026de-eval-within-average-10-nn, lit-cells-networks-20261011-sun2026de-eval-within-average-logistic-regression, lit-cells-networks-20261011-sun2026de-eval-within-average-majority-vote, lit-cells-networks-20261011-sun2026de-eval-within-average-scouter); no evaluation on the protocol is eligible evidence.

Deep learning and foundation models against simple baselines (Ahlmann-Eltze et al. 2025) (Adamson (K562 CRISPRi))

Withheld: a cited source is unchecked or has evidence concerns; a reviewed evaluation is no longer eligible (perturbation-ahlmann-20261011-eval-adamson-gears, perturbation-ahlmann-20261011-eval-adamson-geneformer, perturbation-ahlmann-20261011-eval-adamson-linear-model, perturbation-ahlmann-20261011-eval-adamson-mean, perturbation-ahlmann-20261011-eval-adamson-scbert); no evaluation on the protocol is eligible evidence.

Deep learning and foundation models against simple baselines (Ahlmann-Eltze et al. 2025) (Replogle K562 (CRISPRi))

Withheld: a cited source is unchecked or has evidence concerns; a reviewed evaluation is no longer eligible (perturbation-ahlmann-20261011-eval-replogle-k562-gears, perturbation-ahlmann-20261011-eval-replogle-k562-geneformer, perturbation-ahlmann-20261011-eval-replogle-k562-linear-model, perturbation-ahlmann-20261011-eval-replogle-k562-mean, perturbation-ahlmann-20261011-eval-replogle-k562-scbert); no evaluation on the protocol is eligible evidence.

Deep learning and foundation models against simple baselines (Ahlmann-Eltze et al. 2025) (Replogle RPE1 (CRISPRi))

Withheld: a cited source is unchecked or has evidence concerns; a reviewed evaluation is no longer eligible (perturbation-ahlmann-20261011-eval-replogle-rpe1-gears, perturbation-ahlmann-20261011-eval-replogle-rpe1-geneformer, perturbation-ahlmann-20261011-eval-replogle-rpe1-linear-model, perturbation-ahlmann-20261011-eval-replogle-rpe1-mean, perturbation-ahlmann-20261011-eval-replogle-rpe1-scbert); no evaluation on the protocol is eligible evidence.

Not covered yet

  • The exact Table 6 split manifest, scoring gene subset, scored counts and checkpoint revisions remain unextracted. Do not silently label the MSE endpoint as Top20.
  • The table does not identify its printed spread as SD, SE or a confidence interval; these scores do not establish a statistically supported winner.
  • Both endpoints and the repeated No Perturb row are overlapping evidence from one source comparison, not independent replications.
  • Performance in a new cell system and experimental hit rates need a separate prospective evaluation. Human scientific review remains outstanding.
  • 30 September 2026 audit: A benchmark exists for expression responses, but prospective experiment-selection hit rates and transfer to a new cell system remain uncovered by this mapped comparison.
  • 30 September 2026 audit: Table 6 scoring gene set, checkpoint revisions and exact split manifests remain unextracted; do not infer Top20 MSE or confidence intervals from the printed spreads.
  • Ahlmann-Eltze et al. 2025 (Nature Methods) prints no per-method value. Its per-model means for Fig. 1a, Fig. 2a and Extended Data Fig. 2a and 8a are computed by Rewire from the authors' Source Data rows, not printed. Fig. 2c (bootstrapped ratios to the mean baseline, including the linear models with pretrained embeddings) is not extracted because its aggregation is ambiguous. The single-perturbation Source Data carry evidence concerns: the Fig. 2 legend counts are split 1 only and the Methods counts match no sheet.
  • Control-aware metrics (Systema, Nature Biotechnology 2025) are reported only in figures; no printed per-method values were found in this pass.
  • The one extracted multi-method comparison (Csendes et al. 2025) uses one GEARS perturbation-exclusive split per dataset with no spread; its authors, all employees of one company, built the baselines and re-ran the foundation models.
  • For Norman, Csendes et al. 2025 does not print how many training perturbations are combinations, so its combination results cannot be checked against the requirement for combination examples in GEARS training.
  • PerturBench combination-prediction results are in the store as task records and need a protocol before they can be judged for this use case.

Contribute evidence or propose a correction

How to cite

Assess models for genetic perturbation experiments. Rewire biological model benchmark database, release 2026-11-21-c8bea25e5abb. https://benchmarks.rewirebio.io/use-cases/genetic-perturbation-response/

Release 2026-11-21-c8bea25e5abb. Rankings and values change between releases, so keep the release id in the citation. Cite the original publication as well for any value you reuse: each result links its source.

The compilation and annotations made by Rewire in this release are offered under CC BY 4.0; values and text taken from cited sources stay under the terms of those sources and are not covered by this licence. Data licence

Question, sources and review

Who this is for, setting and exclusions

Choose models and controls for a pilot in the intended cell system before using predictions to select follow-up experiments. The current evidence compares expression prediction; it does not establish experimental hit rates.

Who this is for
  • Researchers designing genetic perturbation screens
  • Computational biologists evaluating perturbation-response models
Biological setting
Research method assessment anchored to the Norman2019 K562 cell-line comparison in GEARS Supplementary Table 6. Both endpoints below describe the same four evaluated configurations.
Research setting
Research use only. These expression endpoints do not establish diagnostic, treatment-selection or patient-response validity.

Outside this use case

  • Cross-cell-type transfer and bulk-sequencing prediction are outside the cited GEARS usage scope.
  • Training GEARS only on single-gene perturbations does not support reliable combination prediction.
  • Expression prediction scores do not establish causal mechanism, fitness effects or successful experimental prioritisation.
Related articles
Sources, review and downloads

Automated source review · 2026-09-30 · Codex source curation, independent worker cross-review and root integration review

Source-backed literature curation with independent automated transcription and scope checks. No new model execution, independent experimental replication, qualified human scientific review or clinical validation.

Release 2026-11-21-c8bea25e5abb · question use-case-genetic-perturbation-response

Use-case input digest bfbb7ab9303becad1e3cb395568c26f41e72ddb67cd1a89a06d9eaa0f76bd95f

Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)