rewirebio.iobenchmarks

Use caseCells and tissuesResearch only

Assess models for genetic perturbation experiments

Which prediction methods and controls should I test before using expression predictions to plan genetic perturbation experiments?

What the evidence shows

The five sources whose comparisons are shown on this page score how well models predict expression changes after genetic perturbations they were not trained on.

Each uses its own split, gene set and metric, so their values are not pooled. Ahlmann-Eltze et al. 2025, who did not develop the deep models and ran them all themselves, scored held-out two-gene CRISPRa perturbations in K562 cells (Norman), training on every single-gene perturbation and half of the pairs. A baseline that adds the two single-gene effects had a mean L2 distance of 2.766 and a Pearson delta of 0.892 over the 1,000 most highly expressed genes, against 3.478 and 0.773 for Geneformer with a linear decoder, 4.096 and 0.631 for GEARS, 4.279 and 0.610 for both UCE and scBERT, 4.616 and 0.527 for scGPT, 4.722 and 0.537 for scFoundation, and 12.205 and 0.146 for CPA; predicting no change gave an L2 distance of 5.616.

Rewire computed these means from the authors' Source Data, no spread is stated, and the UCE and scBERT runs used ground-truth expression of the perturbed genes, a possible test leakage. In PerturBench v4, a preprint whose authors trained every model and built the Linear, Latent Additive and Decoder baselines, the cosine similarity of predicted log fold change on held-out Norman pairs ranged from 0.41 for Biolord and 0.44 for GEARS to 0.79 for the Latent Additive baseline, with CPA at 0.76 and the Linear baseline at 0.60, each a mean over five training seeds. When effects were transferred to melanoma co-culture (Frangieh21) every model scored between 0.01 and 0.22 and every printed DEG recall was 0.000; on held-out cell line and cytokine states (Jiang24) scores ranged from 0.17 for Linear to 0.64 for the Decoder baseline, and the paper describes the held-out states in two different ways.

In Csendes et al. 2025, whose authors are employees of Turbine Ltd. and built the baselines themselves, predicting the mean training response gave a higher Pearson delta than scGPT and scFoundation, as re-run by the same authors, on all four Perturb-seq datasets: 0.711, 0.557, 0.373 and 0.628 for Adamson, Norman, Replogle K562 and Replogle RPE1, against 0.641, 0.554, 0.327 and 0.596 for scGPT and 0.552, 0.459, 0.269 and 0.471 for scFoundation. Each dataset there has one split and one run per model with no spread printed, so small gaps such as 0.557 against 0.554 cannot be tested. The GEARS developers report a mean squared error of 0.216 for GEARS on Norman, against 0.354 for CPA and 0.551 for a no-change control, with spreads of unstated type.

PertEval-scFM, whose authors trained GEARS from scratch, reports an area under its distribution-shift curve of 0.815 for GEARS against 4.484 for an MLP and 4.612 for a mean baseline, lower being better. All of these scores measure agreement with observed expression in held-out perturbations; they do not show which experiments will succeed.

You bring
  • Single-cell expression measurements from perturbed and unperturbed cells, with perturbation and cell-type labels
  • A study design with multiple measured perturbations and cells per perturbation; combinations require combination examples during GEARS training
You want
Predicted post-perturbation expression and changes relative to unperturbed controls, assessed against measured responses.
Evidence in this release
6 comparisons shown · 8 held for review · 29 tools · 42 configurations
Kept up to date
Evidence last checked 9 October 2026 · re-checked monthly, next due 9 November 2026

Baseline

Training mean, no change, additive and linear models. There is no pre-machine-learning method for predicting expression responses; the established comparators are simple statistical controls: the training mean, no change, the sum of single-perturbation effects and a linear model. Benchmarks report that foundation models often do not beat them.

Machine-learning methods are compared with the baseline on the same data in 2 tables: Deep learning and foundation models against simple baselines (Ahlmann-Eltze et al. 2025), Foundation models against simple controls on unseen perturbations (Csendes et al. 2025). Baseline rows are marked in each table.

Tools compared

Comparisons

Each table is one study or protocol with its own truth set and scoring. Compare tools within a table, not across tables.

GEARS Norman2019 CPA-control comparison MSE: Norman2019 perturbation-response MSE

Proxy evidenceAuthor-reported

Mean squared error of predicted versus observed post-perturbation expression; lower is better.

Sorted by MSE (lower is better). The best value in each column is highlighted.

ToolMSE
GEARS0.216 ± 0.053 (best in column)
CPA + KG0.333 ± 0.046
CPA0.354 ± 0.049
No Perturb0.551 ± 0.029

The source does not say whether ± is a standard deviation, a standard error or a range.

Caveats and method
  • Exact Table 6 split and scored gene subset remain unextracted; checkpoint and runtime requirements are not established by these catalogue evaluations.
  • Reported spreads have unspecified type. A numerical difference is not a significance test.
  • Predictions need validation in the intended cell system before experiment selection.

Protocol: GEARS Norman2019 CPA-control comparison MSE: Norman2019 perturbation-response MSE

GEARS Norman2019 CPA-control comparison Pearson DE: Norman2019 perturbation-response Pearson DE

Proxy evidenceAuthor-reported

Pearson correlation of predicted versus observed expression change from unperturbed controls; higher is better.

Sorted by Pearson delta (higher is better). The best value in each column is highlighted.

ToolPearson delta (all genes; mean over perturbations)
GEARS0.556 ± 0.030 (best in column)
CPA + KG0.504 ± 0.029
CPA0.440 ± 0.036
No Perturb0.004 ± 0.006

The source does not say whether ± is a standard deviation, a standard error or a range.

Caveats and method
  • Exact Table 6 split and scored gene subset remain unextracted; checkpoint and runtime requirements are not established by these catalogue evaluations.
  • Reported spreads have unspecified type. A numerical difference is not a significance test.
  • Predictions need validation in the intended cell system before experiment selection.

Protocol: GEARS Norman2019 CPA-control comparison Pearson DE: Norman2019 perturbation-response Pearson DE

PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits

Proxy evidenceAuthor-reported

Area Under the SPECTRA Performance Curve (AUSPC): trapezoidal-rule integral of mean squared error scoring the perturbation-effect delta := P - Xc (Eq. 5, described by the source as a 'log fold change perturbation effect'), across seven increasing-distribution-shift (SPECTRA sparsification) train-test splits; lower is better.

Sorted by AUSPC (lower is better). The best value in each column is highlighted.

ToolAUSPC
GEARS (trained from scratch, no pretrained weights)0.815 ± 0.039 (best in column)
MLP baseline (raw control expression + gene co-expression input, Eq. 3)4.484 ± 0.299
Mean baseline (context mean, no perturbation-specific effect)4.612 ± 0.317

The value after ± or in brackets is a standard error.

Caveats and method
  • AUSPC's printed +/- is the source's own propagated standard error (Appendix F.2, Eqs. F3-F5, propagating per-split triplicate-run MSE uncertainty; main-text Figure 2 caption: 'standard error bars') -- a single, author-reported, propagated quantity, not an independently resampled model-seed standard deviation or confidence interval, and its mathematical derivation is not independently validated here.
  • Per-sparsification-split (S0.1-S0.7) scored perturbation counts are not printed; the paper's Appendix A.1 gives only the eligible raw dataset perturbation count (105 single-gene perturbations), which is not a scored-per-split count and must not be substituted for one.
  • Scores the perturbation-effect delta := P - Xc (Eq. 5), not raw post-perturbation expression; the same proxy endpoint class as the existing Table 6 mappings (expression-change prediction, not prospective experiment-selection hit rate) but a different metric construction from Table 6's own Pearson-delta-expression. Does not establish anything about case 6's intervention-selection endpoint.
  • GEARS here is an independent from-scratch training run by this paper's authors (official implementation, only the train-test split modified, other parameters at default values, no pretrained weights), not the original GEARS authors' own Table 6 checkpoint/run; this is a second, independent GEARS evaluation, not a reproduction of Table 6.
  • The Mean baseline's input population ('all cells in the same context', per the source's own Section 2.2 text) is not fully specified as control-only or training-only; input budgets differ across the three intaken configurations and are not assumed identical.
  • This protocol concerns generalisation to unseen perturbations under distribution shift; it does not test or establish forecasting for unseen cells, donors, or cell-line/context transfer.

Protocol: PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits

PerturBench v4 (Wu et al., NeurIPS 2025 Datasets and Benchmarks)

Proxy evidenceAuthor-reportedIndependent study

Perturbench cosine logfc (higher is better), with the other metrics below each value. The best value in each column is highlighted.

Scroll sideways to see all 3 columns.

ToolNorman19 dual perturbations (K562 CRISPRa)Jiang24 held-out states (genetic perturbations)Frangieh21 co-culture transfer (genetic perturbations)
LA0.79 ± 1 × 10 − 2 (best in column)Perturbench cosine logfc rank 0.005 ± 2 × 10 − 3R 0.000 ± 3 × 10 − 4Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 3.2 ± 6 × 10 − 3Perturbench energy mmd (gene expression space) 6.7 ± 4 × 10 − 3
2 more valuesPerturbench RMSE rank 0.014 ± 1 × 10 − 3Perturbench RMSE of mean 0.043 ± 4 × 10 − 4
0.47 ± 1 × 10 − 3Perturbench cosine logfc rank 0.38 ± 6 × 10 − 3R 0.001 ± 5 × 10 − 4Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 2.6 ± 2 × 10 − 3Perturbench energy mmd (gene expression space) 8.0 ± 9 × 10 − 4
2 more valuesPerturbench RMSE rank 0.38 ± 7 × 10 − 3Perturbench RMSE of mean 0.015 ± 7 × 10 − 5
0.17 ± 6 × 10 − 3Perturbench cosine logfc rank 0.26 ± 1 × 10 − 2R 0.000 ± 0Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 3.1 ± 3 × 10 − 2Perturbench energy mmd (gene expression space) 6.4 ± 3 × 10 − 2
2 more valuesPerturbench RMSE rank 0.21 ± 2 × 10 − 2Perturbench RMSE of mean 0.024 ± 4 × 10 − 4
SAMS-VAE ∗ (S)machine learning0.78 ± 6 × 10 − 3Perturbench cosine logfc rank 0.019 ± 5 × 10 − 3R 0.028 ± 6 × 10 − 3Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 0.74 ± 5 × 10 − 2Perturbench energy mmd (gene expression space) 3.3 ± 5 × 10 − 2
2 more valuesPerturbench RMSE rank 0.030 ± 7 × 10 − 3Perturbench RMSE of mean 0.047 ± 2 × 10 − 3
0.57 ± 5 × 10 − 2Perturbench cosine logfc rank 0.42 ± 2 × 10 − 2R 0.002 ± 1 × 10 − 4Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 1.6 ± 1 × 10 0Perturbench energy mmd (gene expression space) 6.5 ± 2 × 10 0
2 more valuesPerturbench RMSE rank 0.42 ± 3 × 10 − 2Perturbench RMSE of mean 0.016 ± 2 × 10 − 3
0.22 ± 3 × 10 − 3 (best in column)Perturbench cosine logfc rank 0.22 ± 8 × 10 − 3R 0.000 ± 0Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 0.32 ± 4 × 10 − 3Perturbench energy mmd (gene expression space) 2.4 ± 1 × 10 − 2
2 more valuesPerturbench RMSE rank 0.22 ± 1 × 10 − 2Perturbench RMSE of mean 0.025 ± 5 × 10 − 5
CPA* (noAdv)0.77 ± 1 × 10 − 2Perturbench cosine logfc rank 0.0057 ± 3 × 10 − 3R 0.016 ± 9 × 10 − 3Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 2.2 ± 1 × 10 − 1Perturbench energy mmd (gene expression space) 5.5 ± 1 × 10 − 1
2 more valuesPerturbench RMSE rank 0.017 ± 3 × 10 − 3Perturbench RMSE of mean 0.046 ± 1 × 10 − 3
0.60 ± 2 × 10 − 3Perturbench cosine logfc rank 0.39 ± 1 × 10 − 2R 0.005 ± 6 × 10 − 4Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 2.5 ± 3 × 10 − 3Perturbench energy mmd (gene expression space) 8.0 ± 2 × 10 − 3
2 more valuesPerturbench RMSE rank 0.40 ± 1 × 10 − 2Perturbench RMSE of mean 0.015 ± 4 × 10 − 5
0.10 ± 7 × 10 − 3Perturbench cosine logfc rank 0.20 ± 3 × 10 − 3R 0.000 ± 0Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 0.21 ± 2 × 10 − 3Perturbench energy mmd (gene expression space) 1.8 ± 1 × 10 − 2
2 more valuesPerturbench RMSE rank 0.19 ± 3 × 10 − 2Perturbench RMSE of mean 0.027 ± 9 × 10 − 5
LA (scGPT)0.77 ± 4 × 10 − 3Perturbench cosine logfc rank 0.0085 ± 1 × 10 − 3Perturbench RMSE rank 0.013 ± 2 × 10 − 3Perturbench RMSE of mean 0.044 ± 4 × 10 − 4Not reported0.18 ± 6 × 10 − 3Perturbench cosine logfc rank 0.27 ± 1 × 10 − 2Perturbench RMSE rank 0.24 ± 1 × 10 − 2Perturbench RMSE of mean 0.024 ± 6 × 10 − 5
CPA*0.76 ± 4 × 10 − 3Perturbench cosine logfc rank 0.0072 ± 2 × 10 − 3R 0.032 ± 4 × 10 − 3Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 2.2 ± 2 × 10 − 2Perturbench energy mmd (gene expression space) 5.6 ± 3 × 10 − 2
2 more valuesPerturbench RMSE rank 0.019 ± 3 × 10 − 3Perturbench RMSE of mean 0.046 ± 5 × 10 − 4
0.60 ± 7 × 10 − 4Perturbench cosine logfc rank 0.40 ± 9 × 10 − 3R 0.004 ± 2 × 10 − 3Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 2.5 ± 4 × 10 − 3Perturbench energy mmd (gene expression space) 8.0 ± 2 × 10 − 3
2 more valuesPerturbench RMSE rank 0.42 ± 1 × 10 − 2Perturbench RMSE of mean 0.015 ± 5 × 10 − 5
0.10 ± 7 × 10 − 3Perturbench cosine logfc rank 0.30 ± 3 × 10 − 2R 0.000 ± 0Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 0.26 ± 5 × 10 − 3Perturbench energy mmd (gene expression space) 2.1 ± 2 × 10 − 2
2 more valuesPerturbench RMSE rank 0.28 ± 1 × 10 − 2Perturbench RMSE of mean 0.027 ± 1 × 10 − 4
Decoder0.73 ± 2 × 10 − 2Perturbench cosine logfc rank 0.017 ± 6 × 10 − 3Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 3.2 ± 4 × 10 − 3Perturbench energy mmd (gene expression space) 6.7 ± 6 × 10 − 3Perturbench RMSE rank 0.014 ± 4 × 10 − 4Perturbench RMSE of mean 0.043 ± 3 × 10 − 40.64 ± 8 × 10 − 4 (best in column)Perturbench cosine logfc rank 0.32 ± 8 × 10 − 3Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 2.6 ± 2 × 10 − 3Perturbench energy mmd (gene expression space) 8.0 ± 2 × 10 − 3Perturbench RMSE rank 0.32 ± 5 × 10 − 3Perturbench RMSE of mean 0.015 ± 3 × 10 − 50.10 ± 2 × 10 − 3Perturbench cosine logfc rank 0.21 ± 5 × 10 − 3Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 3.3 ± 4 × 10 − 4Perturbench energy mmd (gene expression space) 6.6 ± 8 × 10 − 4Perturbench RMSE rank 0.15 ± 4 × 10 − 4Perturbench RMSE of mean 0.025 ± 4 × 10 − 5
CPA* (scGPT)0.70 ± 2 × 10 − 2Perturbench cosine logfc rank 0.025 ± 6 × 10 − 3Perturbench RMSE rank 0.036 ± 3 × 10 − 3Perturbench RMSE of mean 0.053 ± 1 × 10 − 3Not reported0.07 ± 4 × 10 − 3Perturbench cosine logfc rank 0.26 ± 5 × 10 − 3Perturbench RMSE rank 0.26 ± 1 × 10 − 2Perturbench RMSE of mean 0.029 ± 2 × 10 − 4
Linear0.60 ± 2 × 10 − 2Perturbench cosine logfc rank 0.035 ± 4 × 10 − 3R 0.018 ± 2 × 10 − 3Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 1.2 ± 4 × 10 − 2Perturbench energy mmd (gene expression space) 2.5 ± 4 × 10 − 2
2 more valuesPerturbench RMSE rank 0.016 ± 8 × 10 − 4Perturbench RMSE of mean 0.057 ± 3 × 10 − 3
0.17 ± 8 × 10 − 5Perturbench cosine logfc rank 0.34 ± 2 × 10 − 3R 0.003 ± 2 × 10 − 4Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 1.3 ± 2 × 10 − 3Perturbench energy mmd (gene expression space) 3.1 ± 2 × 10 − 3
2 more valuesPerturbench RMSE rank 0.43 ± 1 × 10 − 3Perturbench RMSE of mean 0.038 ± 5 × 10 − 5
0.01 ± 4 × 10 − 4Perturbench cosine logfc rank 0.24 ± 9 × 10 − 4R 0.000 ± 0Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 0.79 ± 6 × 10 − 3Perturbench energy mmd (gene expression space) 2.1 ± 8 × 10 − 3
2 more valuesPerturbench RMSE rank 0.30 ± 2 × 10 − 3Perturbench RMSE of mean 0.043 ± 7 × 10 − 5
SAMS-VAE*0.45 ± 2 × 10 − 2Perturbench cosine logfc rank 0.021 ± 5 × 10 − 3R 0.000 ± 0Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 1.9 ± 3 × 10 − 2Perturbench energy mmd (gene expression space) 4.1 ± 4 × 10 − 2
2 more valuesPerturbench RMSE rank 0.026 ± 2 × 10 − 3Perturbench RMSE of mean 0.084 ± 7 × 10 − 4
0.59 ± 3 × 10 − 3Perturbench cosine logfc rank 0.45 ± 9 × 10 − 3R 0.000 ± 4 × 10 − 5Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 0.32 ± 5 × 10 − 3Perturbench energy mmd (gene expression space) 4.5 ± 7 × 10 − 3
2 more valuesPerturbench RMSE rank 0.45 ± 6 × 10 − 3Perturbench RMSE of mean 0.017 ± 2 × 10 − 4
0.15 ± 2 × 10 − 2Perturbench cosine logfc rank 0.48 ± 2 × 10 − 2R 0.000 ± 0Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 0.29 ± 2 × 10 − 3Perturbench energy mmd (gene expression space) 2.2 ± 1 × 10 − 2
2 more valuesPerturbench RMSE rank 0.46 ± 2 × 10 − 2Perturbench RMSE of mean 0.026 ± 2 × 10 − 4
GEARSmachine learning0.44 ± 5 × 10 − 3Perturbench cosine logfc rank 0.051 ± 1 × 10 − 2Perturbench RMSE rank 0.055 ± 6 × 10 − 3Perturbench RMSE of mean 0.069 ± 1 × 10 − 3Not reportedNot reported
Biolord*0.41 ± 2 × 10 − 2Perturbench cosine logfc rank 0.027 ± 1 × 10 − 3R 0.000 ± 0Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 1.6 ± 5 × 10 − 3Perturbench energy mmd (gene expression space) 2.8 ± 5 × 10 − 3
2 more valuesPerturbench RMSE rank 0.028 ± 1 × 10 − 3Perturbench RMSE of mean 0.086 ± 6 × 10 − 4
Not reported0.12 ± 9 × 10 − 3Perturbench cosine logfc rank 0.29 ± 3 × 10 − 2R 0.000 ± 1 × 10 − 4Perturbench energy mmd (PCA space: top 256 principal components fitted on the ground-truth test split) 0.20 ± 3 × 10 − 3Perturbench energy mmd (gene expression space) 1.1 ± 2 × 10 − 2
2 more valuesPerturbench RMSE rank 0.21 ± 4 × 10 − 2Perturbench RMSE of mean 0.027 ± 2 × 10 − 4

R recall. ± is a standard deviation.

Caveats and method

What each column measures:

  • Norman19 dual perturbations (K562 CRISPRa): Cosine similarity and rank of predicted log fold change, RMSE and RMSE rank of the mean expression profile, MMD and DEG recall for held-out K562 CRISPRa dual-gene perturbations (Norman19), for CPA, SAMS-VAE, Biolord, GEARS, Latent Additive (with and without scGPT embeddings), Decoder and Linear models
  • Jiang24 held-out states (genetic perturbations): Cosine similarity and rank of predicted log fold change, MMD, DEG recall and RMSE for held-out genetic perturbations in held-out cell line and cytokine states (Jiang24)
  • Frangieh21 co-culture transfer (genetic perturbations): Cosine similarity and rank of predicted log fold change, MMD, DEG recall and RMSE for held-out genetic perturbations in melanoma co-culture, trained on all other perturbations, including the other culture conditions and the remaining 30% of co-culture perturbations (Frangieh21)
  • The PerturBench authors trained and tuned every model themselves and developed the Linear, Latent Additive and Decoder baselines; CPA, SAMS-VAE, Biolord and GEARS are their reimplementations or wrappers.
  • Preprint (arXiv v4 of the NeurIPS 2025 Datasets and Benchmarks paper); the proceedings version was not compared.
  • Spread is one standard deviation over 5 training seeds of one HPO-selected configuration, not over data splits.
  • Expression-prediction scores do not establish causal mechanism or experimental hit rates (use-case exclusion).
  • GEARS rows have no MMD or DEG recall (printed '-').
  • Table 4 caption (9 of 30 states held out) and Appendix D.4 (12 cytokine-cell line combinations) describe the held-out states differently.
  • The use case lists cross-cell-type transfer as outside the cited GEARS usage scope; this protocol tests transfer to held-out cell line and cytokine states and does not include GEARS, so it bears only on the use case's open question about performance in a new cell system.

Protocols: PerturBench v4 combination prediction of held-out dual perturbations on Norman19, PerturBench v4 covariate transfer to held-out perturbations in held-out biological states on Jiang24, PerturBench v4 covariate transfer from mono-culture to co-culture on Frangieh21

Deep learning and foundation models against simple baselines (Ahlmann-Eltze et al. 2025)

Proxy evidenceAuthor-reportedIndependent study

Mean L2 distance and mean Pearson delta between predicted and observed expression over the 1,000 most highly expressed genes, for scGPT, scFoundation, GEARS, CPA and three foundation models with a linear decoder (Geneformer, UCE, scBERT) against the no change and additive baselines, Norman doubles (K562 CRISPRa)

L2 distance (lower is better), with the other metrics below each value. The best value in each column is highlighted.

ToolNorman doubles (K562 CRISPRa)
Additive (Source Data method "additive_model")Baselineconventional2.766Computed by Rewire from the authors' source data (best in column)Pearson delta 0.892
Geneformer (Source Data method "geneformer")foundation model3.478Computed by Rewire from the authors' source dataPearson delta 0.773
GEARS (Source Data method "gears")machine learning4.096Computed by Rewire from the authors' source dataPearson delta 0.631
scBERT (Source Data method "scbert")foundation model4.279Computed by Rewire from the authors' source dataPearson delta 0.610
UCE (Source Data method "uce")foundation model4.279Computed by Rewire from the authors' source dataPearson delta 0.610
scGPT (Source Data method "scgpt")foundation model4.616Computed by Rewire from the authors' source dataPearson delta 0.527
scFoundation (Source Data method "scfoundation")foundation model4.722Computed by Rewire from the authors' source dataPearson delta 0.537
No change (Source Data method "no_change")Baselineconventional5.616Computed by Rewire from the authors' source data
CPA (Source Data method "cpa")machine learning12.205Computed by Rewire from the authors' source dataPearson delta 0.146
Caveats and method
  • Values are computed by Rewire from the authors' per-perturbation Source Data (attributes.derivation on each result); the paper prints no per-model number.
  • The authors ran all models themselves; the deep models were not run by their developers, although the developers were asked to review the code.
  • No spread of the mean is stated, so small differences between models are not testable from these values.
  • Training includes every single perturbation and half of the doubles, the setting the use case's GEARS combination guidance requires.
  • UCE and scBERT post-perturbation embeddings use ground-truth expression of the perturbed genes (possible test leakage, Methods).
  • The Pearson delta for the no change baseline is undefined and not recorded.
  • The Norman gene count differs between the main text (19,264) and the Extended Data Fig. 10 legend (19,624).

Protocol: Norman double perturbations: expression prediction for held-out double perturbations (Ahlmann-Eltze et al. 2025 Fig. 1a and Extended Data Fig. 2a)

Foundation models against simple controls on unseen perturbations (Csendes et al. 2025)

Proxy evidenceAuthor-reportedIndependent study

Pearson delta (higher is better), with the other metrics below each value. The best value in each column is highlighted.

Scroll sideways to see all 4 columns.

ToolAdamson (K562 CRISPRi)Norman (K562 CRISPRa, singles and pairs)Replogle K562 (CRISPRi)Replogle RPE1 (CRISPRi)
KNN_scGPTconventional0.753 (best in column)Pearson (pseudo-bulk, all genes) 0.994Pearson (top 20 DE genes (t-test, GEARS list)) 0.976Pearson delta (t-test, GEARS list) 0.778Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.834Pearson delta (Wilcoxon test) 0.7820.603Pearson (pseudo-bulk, all genes) 0.989Pearson (top 20 DE genes (t-test, GEARS list)) 0.935Pearson delta (t-test, GEARS list) 0.643Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.681Pearson delta (Wilcoxon test) 0.7050.374Pearson (pseudo-bulk, all genes) 0.987Pearson (top 20 DE genes (t-test, GEARS list)) 0.959Pearson delta (t-test, GEARS list) 0.476Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.503Pearson delta (Wilcoxon test) 0.4940.5805Pearson (pseudo-bulk, all genes) 0.975Pearson (top 20 DE genes (t-test, GEARS list)) 0.911Pearson delta (t-test, GEARS list) 0.590Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.598Pearson delta (Wilcoxon test) 0.680
RF_goconventional0.739Pearson (pseudo-bulk, all genes) 0.993Pearson (top 20 DE genes (t-test, GEARS list)) 0.971Pearson delta (t-test, GEARS list) 0.742Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.796Pearson delta (Wilcoxon test) 0.7340.586Pearson (pseudo-bulk, all genes) 0.988Pearson (top 20 DE genes (t-test, GEARS list)) 0.916Pearson delta (t-test, GEARS list) 0.669Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.702Pearson delta (Wilcoxon test) 0.6880.480 (best in column)Pearson (pseudo-bulk, all genes) 0.989Pearson (top 20 DE genes (t-test, GEARS list)) 0.966Pearson delta (t-test, GEARS list) 0.597Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.623Pearson delta (Wilcoxon test) 0.6240.6477Pearson (pseudo-bulk, all genes) 0.979Pearson (top 20 DE genes (t-test, GEARS list)) 0.915Pearson delta (t-test, GEARS list) 0.670Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.673Pearson delta (Wilcoxon test) 0.749
RF_scGPTconventional0.727Pearson (pseudo-bulk, all genes) 0.993Pearson (top 20 DE genes (t-test, GEARS list)) 0.970Pearson delta (t-test, GEARS list) 0.735Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.790Pearson delta (Wilcoxon test) 0.7320.583Pearson (pseudo-bulk, all genes) 0.988Pearson (top 20 DE genes (t-test, GEARS list)) 0.914Pearson delta (t-test, GEARS list) 0.647Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.685Pearson delta (Wilcoxon test) 0.7100.421Pearson (pseudo-bulk, all genes) 0.988Pearson (top 20 DE genes (t-test, GEARS list)) 0.958Pearson delta (t-test, GEARS list) 0.520Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.545Pearson delta (Wilcoxon test) 0.5380.6347Pearson (pseudo-bulk, all genes) 0.978Pearson (top 20 DE genes (t-test, GEARS list)) 0.918Pearson delta (t-test, GEARS list) 0.646Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.647Pearson delta (Wilcoxon test) 0.725
RF_scFoundationconventional0.720Pearson (pseudo-bulk, all genes) 0.993Pearson (top 20 DE genes (t-test, GEARS list)) 0.970Pearson delta (t-test, GEARS list) 0.730Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.785Pearson delta (Wilcoxon test) 0.7250.623Pearson (pseudo-bulk, all genes) 0.988Pearson (top 20 DE genes (t-test, GEARS list)) 0.917Pearson delta (t-test, GEARS list) 0.703Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.745Pearson delta (Wilcoxon test) 0.7430.359Pearson (pseudo-bulk, all genes) 0.987Pearson (top 20 DE genes (t-test, GEARS list)) 0.955Pearson delta (t-test, GEARS list) 0.447Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.470Pearson delta (Wilcoxon test) 0.4690.6266Pearson (pseudo-bulk, all genes) 0.976Pearson (top 20 DE genes (t-test, GEARS list)) 0.919Pearson delta (t-test, GEARS list) 0.638Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.638Pearson delta (Wilcoxon test) 0.716
EN_scGPTconventional0.718Pearson (pseudo-bulk, all genes) 0.993Pearson (top 20 DE genes (t-test, GEARS list)) 0.972Pearson delta (t-test, GEARS list) 0.738Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.793Pearson delta (Wilcoxon test) 0.7370.614Pearson (pseudo-bulk, all genes) 0.988Pearson (top 20 DE genes (t-test, GEARS list)) 0.924Pearson delta (t-test, GEARS list) 0.681Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.720Pearson delta (Wilcoxon test) 0.7290.376Pearson (pseudo-bulk, all genes) 0.987Pearson (top 20 DE genes (t-test, GEARS list)) 0.955Pearson delta (t-test, GEARS list) 0.472Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.498Pearson delta (Wilcoxon test) 0.4890.6288Pearson (pseudo-bulk, all genes) 0.976Pearson (top 20 DE genes (t-test, GEARS list)) 0.920Pearson delta (t-test, GEARS list) 0.641Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.641Pearson delta (Wilcoxon test) 0.717
KNN_goconventional0.713Pearson (pseudo-bulk, all genes) 0.992Pearson (top 20 DE genes (t-test, GEARS list)) 0.971Pearson delta (t-test, GEARS list) 0.745Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.802Pearson delta (Wilcoxon test) 0.7410.618Pearson (pseudo-bulk, all genes) 0.989Pearson (top 20 DE genes (t-test, GEARS list)) 0.931Pearson delta (t-test, GEARS list) 0.681Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.717Pearson delta (Wilcoxon test) 0.7110.435Pearson (pseudo-bulk, all genes) 0.988Pearson (top 20 DE genes (t-test, GEARS list)) 0.966Pearson delta (t-test, GEARS list) 0.550Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.577Pearson delta (Wilcoxon test) 0.5750.6237Pearson (pseudo-bulk, all genes) 0.977Pearson (top 20 DE genes (t-test, GEARS list)) 0.909Pearson delta (t-test, GEARS list) 0.660Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.664Pearson delta (Wilcoxon test) 0.746
EN_goconventional0.711Pearson (pseudo-bulk, all genes) 0.992Pearson (top 20 DE genes (t-test, GEARS list)) 0.972Pearson delta (t-test, GEARS list) 0.751Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.805Pearson delta (Wilcoxon test) 0.7530.535Pearson (pseudo-bulk, all genes) 0.986Pearson (top 20 DE genes (t-test, GEARS list)) 0.917Pearson delta (t-test, GEARS list) 0.620Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.636Pearson delta (Wilcoxon test) 0.6190.430Pearson (pseudo-bulk, all genes) 0.988Pearson (top 20 DE genes (t-test, GEARS list)) 0.960Pearson delta (t-test, GEARS list) 0.533Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.558Pearson delta (Wilcoxon test) 0.5530.6383Pearson (pseudo-bulk, all genes) 0.978Pearson (top 20 DE genes (t-test, GEARS list)) 0.919Pearson delta (t-test, GEARS list) 0.669Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.673Pearson delta (Wilcoxon test) 0.740
meanBaselineconventional0.711Pearson (pseudo-bulk, all genes) 0.993Pearson (top 20 DE genes (t-test, GEARS list)) 0.969Pearson delta (t-test, GEARS list) 0.729Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.785Pearson delta (Wilcoxon test) 0.7240.557Pearson (pseudo-bulk, all genes) 0.986Pearson (top 20 DE genes (t-test, GEARS list)) 0.889Pearson delta (t-test, GEARS list) 0.642Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.682Pearson delta (Wilcoxon test) 0.6850.373Pearson (pseudo-bulk, all genes) 0.987Pearson (top 20 DE genes (t-test, GEARS list)) 0.954Pearson delta (t-test, GEARS list) 0.460Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.484Pearson delta (Wilcoxon test) 0.4820.6284Pearson (pseudo-bulk, all genes) 0.976Pearson (top 20 DE genes (t-test, GEARS list)) 0.920Pearson delta (t-test, GEARS list) 0.640Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.640Pearson delta (Wilcoxon test) 0.716
EN_scElmoconventional0.711Pearson (pseudo-bulk, all genes) 0.993Pearson (top 20 DE genes (t-test, GEARS list)) 0.969Pearson delta (t-test, GEARS list) 0.731Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.786Pearson delta (Wilcoxon test) 0.7240.580Pearson (pseudo-bulk, all genes) 0.987Pearson (top 20 DE genes (t-test, GEARS list)) 0.903Pearson delta (t-test, GEARS list) 0.678Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.719Pearson delta (Wilcoxon test) 0.7160.375Pearson (pseudo-bulk, all genes) 0.987Pearson (top 20 DE genes (t-test, GEARS list)) 0.955Pearson delta (t-test, GEARS list) 0.469Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.493Pearson delta (Wilcoxon test) 0.4900.6284Pearson (pseudo-bulk, all genes) 0.976Pearson (top 20 DE genes (t-test, GEARS list)) 0.920Pearson delta (t-test, GEARS list) 0.640Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.640Pearson delta (Wilcoxon test) 0.716
RF_scElmoconventional0.706Pearson (pseudo-bulk, all genes) 0.992Pearson (top 20 DE genes (t-test, GEARS list)) 0.968Pearson delta (t-test, GEARS list) 0.726Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.781Pearson delta (Wilcoxon test) 0.7200.663 (best in column)Pearson (pseudo-bulk, all genes) 0.990Pearson (top 20 DE genes (t-test, GEARS list)) 0.928Pearson delta (t-test, GEARS list) 0.750Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.793Pearson delta (Wilcoxon test) 0.7880.471Pearson (pseudo-bulk, all genes) 0.989Pearson (top 20 DE genes (t-test, GEARS list)) 0.963Pearson delta (t-test, GEARS list) 0.598Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.630Pearson delta (Wilcoxon test) 0.5990.6506 (best in column)Pearson (pseudo-bulk, all genes) 0.980Pearson (top 20 DE genes (t-test, GEARS list)) 0.919Pearson delta (t-test, GEARS list) 0.666Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.667Pearson delta (Wilcoxon test) 0.743
EN_scFoundationconventional0.694Pearson (pseudo-bulk, all genes) 0.992Pearson (top 20 DE genes (t-test, GEARS list)) 0.966Pearson delta (t-test, GEARS list) 0.714Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.769Pearson delta (Wilcoxon test) 0.7060.600Pearson (pseudo-bulk, all genes) 0.988Pearson (top 20 DE genes (t-test, GEARS list)) 0.930Pearson delta (t-test, GEARS list) 0.685Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.723Pearson delta (Wilcoxon test) 0.7040.373Pearson (pseudo-bulk, all genes) 0.987Pearson (top 20 DE genes (t-test, GEARS list)) 0.954Pearson delta (t-test, GEARS list) 0.461Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.484Pearson delta (Wilcoxon test) 0.4820.6284Pearson (pseudo-bulk, all genes) 0.976Pearson (top 20 DE genes (t-test, GEARS list)) 0.920Pearson delta (t-test, GEARS list) 0.640Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.640Pearson delta (Wilcoxon test) 0.716
KNN_scElmoconventional0.642Pearson (pseudo-bulk, all genes) 0.990Pearson (top 20 DE genes (t-test, GEARS list)) 0.964Pearson delta (t-test, GEARS list) 0.651Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.706Pearson delta (Wilcoxon test) 0.6710.638Pearson (pseudo-bulk, all genes) 0.989Pearson (top 20 DE genes (t-test, GEARS list)) 0.935Pearson delta (t-test, GEARS list) 0.730Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.768Pearson delta (Wilcoxon test) 0.7560.453Pearson (pseudo-bulk, all genes) 0.989Pearson (top 20 DE genes (t-test, GEARS list)) 0.968Pearson delta (t-test, GEARS list) 0.577Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.609Pearson delta (Wilcoxon test) 0.5760.6202Pearson (pseudo-bulk, all genes) 0.979Pearson (top 20 DE genes (t-test, GEARS list)) 0.911Pearson delta (t-test, GEARS list) 0.639Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.646Pearson delta (Wilcoxon test) 0.729
scGPTfoundation model0.641Pearson (pseudo-bulk, all genes) 0.990Pearson (top 20 DE genes (t-test, GEARS list)) 0.979Pearson delta (t-test, GEARS list) 0.797Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.7950.554Pearson (pseudo-bulk, all genes) 0.988Pearson (top 20 DE genes (t-test, GEARS list)) 0.935Pearson delta (t-test, GEARS list) 0.732Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.7000.327Pearson (pseudo-bulk, all genes) 0.986Pearson (top 20 DE genes (t-test, GEARS list)) 0.960Pearson delta (t-test, GEARS list) 0.481Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.4420.5964Pearson (pseudo-bulk, all genes) 0.974Pearson (top 20 DE genes (t-test, GEARS list)) 0.924Pearson delta (t-test, GEARS list) 0.652Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.629
KNN_scFoundationconventional0.629Pearson (pseudo-bulk, all genes) 0.990Pearson (top 20 DE genes (t-test, GEARS list)) 0.960Pearson delta (t-test, GEARS list) 0.698Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.747Pearson delta (Wilcoxon test) 0.6940.576Pearson (pseudo-bulk, all genes) 0.988Pearson (top 20 DE genes (t-test, GEARS list)) 0.931Pearson delta (t-test, GEARS list) 0.629Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.658Pearson delta (Wilcoxon test) 0.6770.265Pearson (pseudo-bulk, all genes) 0.985Pearson (top 20 DE genes (t-test, GEARS list)) 0.950Pearson delta (t-test, GEARS list) 0.328Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.344Pearson delta (Wilcoxon test) 0.3470.5775Pearson (pseudo-bulk, all genes) 0.972Pearson (top 20 DE genes (t-test, GEARS list)) 0.911Pearson delta (t-test, GEARS list) 0.600Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.604Pearson delta (Wilcoxon test) 0.688
scFoundationfoundation model0.552Pearson (pseudo-bulk, all genes) 0.985Pearson (top 20 DE genes (t-test, GEARS list)) 0.969Pearson delta (t-test, GEARS list) 0.719Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.7800.459Pearson (pseudo-bulk, all genes) 0.966Pearson (top 20 DE genes (t-test, GEARS list)) 0.924Pearson delta (t-test, GEARS list) 0.705Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.7100.269Pearson (pseudo-bulk, all genes) 0.977Pearson (top 20 DE genes (t-test, GEARS list)) 0.952Pearson delta (t-test, GEARS list) 0.383Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.3940.4709Pearson (pseudo-bulk, all genes) 0.971Pearson (top 20 DE genes (t-test, GEARS list)) 0.912Pearson delta (t-test, GEARS list) 0.495Pearson delta ((t-test) excluding the CRISPR target gene(s)) 0.490
Caveats and method

What each column measures:

  • Adamson (K562 CRISPRi): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Adamson (K562 CRISPRi)
  • Norman (K562 CRISPRa, singles and pairs): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Norman (K562 CRISPRa, singles and pairs)
  • Replogle K562 (CRISPRi): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Replogle K562 (CRISPRi)
  • Replogle RPE1 (CRISPRi): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Replogle RPE1 (CRISPRi)
  • Datasets have low perturbation-specific variance (Results, second section), so differences between models are small.
  • Top-20 DE metrics include the CRISPR target gene unless stated, which favours scGPT (claim).
  • All authors are full-time employees of Turbine Ltd., and one is a founder (Competing interests). They built the Train Mean and regressor baselines, configured and ran the scGPT and scFoundation fine-tuning themselves, and argue that the baselines win; the foundation models were not run by their developers, so their configuration may not be the best available.
  • One GEARS perturbation-exclusive split per dataset, within one cell line, with one fine-tuning run per foundation model (best epoch on validation); no seeds, repeats or spread are printed, so small differences between models are not testable.
  • The Wilcoxon top-20 DE metric is blank for scGPT and scFoundation in Supplementary Table 2.
  • Results paragraphs 7 and 8 cite Supplementary Table 1 for values that are in Supplementary Table 2.
  • All perturbations are single-gene CRISPRi in one cell line, so the use case's combination-training and cross-cell-type exclusions are not engaged.
  • 79 of the 116 test perturbations are two-gene combinations (9 with neither gene seen in training, 52 with one, 18 with both) and 37 are unseen single genes (Supplementary Table 1); every value pools them. The per-subgroup values (Supplementary Table 3) are not extracted because their metric is not labelled.
  • The make-up of the 138 training perturbations is not printed, so whether training included combination examples, which the use case requires for GEARS combination prediction, cannot be confirmed from this source. scFoundation runs through GEARS, so the use case's GEARS combination exclusion applies to it directly.
  • The regressors represent a combination as the sum of the two genes' embeddings (Methods 'Baseline models').

Protocols: Adamson et al. 2016 Perturb-seq, K562 CRISPRi single perturbations: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2), Norman et al. 2019 Perturb-seq, K562 CRISPRa single and double perturbations: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2), Replogle et al. 2022 genome-wide Perturb-seq subset, K562 CRISPRi: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2), Replogle et al. 2022 genome-wide Perturb-seq subset, RPE1 CRISPRi: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2)

Held for review

Recorded but not shown as evidence until the reason below is resolved.

Harmonised single-cell and spatial foundation model benchmark (Chen et al. 2026) (Adamson K562 Perturb-seq)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-chen2026-eval-perturbation-adamson-cellplm, lit-cells-networks-20261011-chen2026-eval-perturbation-adamson-genept-w, lit-cells-networks-20261011-chen2026-eval-perturbation-adamson-scelmo, lit-cells-networks-20261011-chen2026-eval-perturbation-adamson-scgpt-spatial); no evaluation on the protocol is eligible evidence.

Harmonised single-cell and spatial foundation model benchmark (Chen et al. 2026) (Norman K562 Perturb-seq)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-chen2026-eval-perturbation-norman-cellplm, lit-cells-networks-20261011-chen2026-eval-perturbation-norman-genept-w, lit-cells-networks-20261011-chen2026-eval-perturbation-norman-scelmo, lit-cells-networks-20261011-chen2026-eval-perturbation-norman-scgpt-spatial); no evaluation on the protocol is eligible evidence.

Harmonised single-cell and spatial foundation model benchmark (Chen et al. 2026) (Replogle K562 Perturb-seq)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-chen2026-eval-perturbation-replogle-cellplm, lit-cells-networks-20261011-chen2026-eval-perturbation-replogle-genept-w, lit-cells-networks-20261011-chen2026-eval-perturbation-replogle-scelmo, lit-cells-networks-20261011-chen2026-eval-perturbation-replogle-scgpt-spatial); no evaluation on the protocol is eligible evidence.

DE classification benchmark (Sun et al. 2026) (Held-out cell line)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-sun2026de-eval-cross-average-1-nn, lit-cells-networks-20261011-sun2026de-eval-cross-average-10-nn, lit-cells-networks-20261011-sun2026de-eval-cross-average-logistic-regression, lit-cells-networks-20261011-sun2026de-eval-cross-average-majority-vote, lit-cells-networks-20261011-sun2026de-eval-cross-average-scvi-mean); no evaluation on the protocol is eligible evidence.

DE classification benchmark (Sun et al. 2026) (Unseen perturbations within a cell line)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-sun2026de-eval-within-average-1-nn, lit-cells-networks-20261011-sun2026de-eval-within-average-10-nn, lit-cells-networks-20261011-sun2026de-eval-within-average-logistic-regression, lit-cells-networks-20261011-sun2026de-eval-within-average-majority-vote, lit-cells-networks-20261011-sun2026de-eval-within-average-scouter); no evaluation on the protocol is eligible evidence.

Deep learning and foundation models against simple baselines (Ahlmann-Eltze et al. 2025) (Adamson (K562 CRISPRi))

Withheld: a cited source is unchecked or has evidence concerns; a reviewed evaluation is no longer eligible (perturbation-ahlmann-20261011-eval-adamson-gears, perturbation-ahlmann-20261011-eval-adamson-geneformer, perturbation-ahlmann-20261011-eval-adamson-linear-model, perturbation-ahlmann-20261011-eval-adamson-mean, perturbation-ahlmann-20261011-eval-adamson-scbert); no evaluation on the protocol is eligible evidence.

Deep learning and foundation models against simple baselines (Ahlmann-Eltze et al. 2025) (Replogle K562 (CRISPRi))

Withheld: a cited source is unchecked or has evidence concerns; a reviewed evaluation is no longer eligible (perturbation-ahlmann-20261011-eval-replogle-k562-gears, perturbation-ahlmann-20261011-eval-replogle-k562-geneformer, perturbation-ahlmann-20261011-eval-replogle-k562-linear-model, perturbation-ahlmann-20261011-eval-replogle-k562-mean, perturbation-ahlmann-20261011-eval-replogle-k562-scbert); no evaluation on the protocol is eligible evidence.

Deep learning and foundation models against simple baselines (Ahlmann-Eltze et al. 2025) (Replogle RPE1 (CRISPRi))

Withheld: a cited source is unchecked or has evidence concerns; a reviewed evaluation is no longer eligible (perturbation-ahlmann-20261011-eval-replogle-rpe1-gears, perturbation-ahlmann-20261011-eval-replogle-rpe1-geneformer, perturbation-ahlmann-20261011-eval-replogle-rpe1-linear-model, perturbation-ahlmann-20261011-eval-replogle-rpe1-mean, perturbation-ahlmann-20261011-eval-replogle-rpe1-scbert); no evaluation on the protocol is eligible evidence.

Not covered yet

  • The exact Table 6 split manifest, scoring gene subset, scored counts and checkpoint revisions remain unextracted. Do not silently label the MSE endpoint as Top20.
  • The table does not identify its printed spread as SD, SE or a confidence interval; these scores do not establish a statistically supported winner.
  • Both endpoints and the repeated No Perturb row are overlapping evidence from one source comparison, not independent replications.
  • Performance in a new cell system and experimental hit rates need a separate prospective evaluation. Human scientific review remains outstanding.
  • 30 September 2026 audit: A benchmark exists for expression responses, but prospective experiment-selection hit rates and transfer to a new cell system remain uncovered by this mapped comparison.
  • 30 September 2026 audit: Table 6 scoring gene set, checkpoint revisions and exact split manifests remain unextracted; do not infer Top20 MSE or confidence intervals from the printed spreads.
  • Ahlmann-Eltze et al. 2025 (Nature Methods) prints no per-method value. Its per-model means for Fig. 1a, Fig. 2a and Extended Data Fig. 2a and 8a are computed by Rewire from the authors' Source Data rows, not printed. Fig. 2c (bootstrapped ratios to the mean baseline, including the linear models with pretrained embeddings) is not extracted because its aggregation is ambiguous. The single-perturbation Source Data carry evidence concerns: the Fig. 2 legend counts are split 1 only and the Methods counts match no sheet.
  • Control-aware metrics (Systema, Nature Biotechnology 2025) are reported only in figures; no printed per-method values were found in this pass.
  • The one extracted multi-method comparison (Csendes et al. 2025) uses one GEARS perturbation-exclusive split per dataset with no spread; its authors, all employees of one company, built the baselines and re-ran the foundation models.
  • For Norman, Csendes et al. 2025 does not print how many training perturbations are combinations, so its combination results cannot be checked against the requirement for combination examples in GEARS training.
  • PerturBench combination-prediction results are in the store as task records and need a protocol before they can be judged for this use case.

Contribute evidence or propose a correction

Question, sources and review

Who this is for, setting and exclusions

Choose models and controls for a pilot in the intended cell system before using predictions to select follow-up experiments. The current evidence compares expression prediction; it does not establish experimental hit rates.

Who this is for
  • Researchers designing genetic perturbation screens
  • Computational biologists evaluating perturbation-response models
Biological setting
Research method assessment anchored to the Norman2019 K562 cell-line comparison in GEARS Supplementary Table 6. Both endpoints below describe the same four evaluated configurations.
Research setting
Research use only. These expression endpoints do not establish diagnostic, treatment-selection or patient-response validity.

Outside this use case

  • Cross-cell-type transfer and bulk-sequencing prediction are outside the cited GEARS usage scope.
  • Training GEARS only on single-gene perturbations does not support reliable combination prediction.
  • Expression prediction scores do not establish causal mechanism, fitness effects or successful experimental prioritisation.
Related articles

Read the background, methods and practical considerations behind this question.

Sources, review and downloads

Automated source review · 2026-09-30 · Codex source curation, independent worker cross-review and root integration review

Source-backed literature curation with independent automated transcription and scope checks. No new model execution, independent experimental replication, qualified human scientific review or clinical validation.

Release 2026-11-02-a7f03c9e55a5 · question use-case-genetic-perturbation-response

Use-case input digest 43a80aef5473f1a49c013b6d46b720b850ff5a0d18dcbb741dcdeed1ed082db1

Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)