Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 2 models
• 3 methods
• 11 configurations
• 4 protocols
• 4 datasets
• 30 evaluations
• 59 results
• 15 evidence claims

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• Baseline: Assess models for genetic perturbation experiments: Baseline Training mean, no change, additive and linear models (perturbation-response-20261009-source-csendes2025: Abstract and Results 'Benchmarking of post-perturbation RNA-seq prediction methods' (Train Mean); perturbation-ahlmann-20261011-source-article: title and Results (no change, additive and linear baselines))
• design_limitation: perturbation-ahlmann-20261011-config-uce: Design limitation UCE has no in silico perturbation, so the authors took the unperturbed expression matrix and overwrote the rows of the perturbed genes with ground-truth values, accepting that this test data leakage could in theory give UCE an advantage. scBERT's perturbed embedding was computed the same way. (Methods 'Software versions and parameters', UCE and scBERT paragraphs)
• Evidence summary: genetic perturbation response prediction: Summary The five sources whose comparisons are shown on this page score how well models predict expression changes after genetic perturbations they were not trained on. Each uses its own split, gene set and metric, so their values are not pooled. Ahlmann-Eltze et al. 2025, who did not develop the deep models and ran them all themselves, scored held-out two-gene CRISPRa perturbations in K562 cells (Norman), training on every single-gene perturbation and half of the pairs. A baseline that adds the two single-gene effects had a mean L2 distance of 2.766 and a Pearson delta of 0.892 over the 1,000 most highly expressed genes, against 3.478 and 0.773 for Geneformer with a linear decoder, 4.096 and 0.631 for GEARS, 4.279 and 0.610 for both UCE and scBERT, 4.616 and 0.527 for scGPT, 4.722 and 0.537 for scFoundation, and 12.205 and 0.146 for CPA; predicting no change gave an L2 distance of 5.616. Rewire computed these means from the authors' Source Data, no spread is stated, and the UCE and scBERT runs used ground-truth expression of the perturbed genes, a possible test leakage. In PerturBench v4, a preprint whose authors trained every model and built the Linear, Latent Additive and Decoder baselines, the cosine similarity of predicted log fold change on held-out Norman pairs ranged from 0.41 for Biolord and 0.44 for GEARS to 0.79 for the Latent Additive baseline, with CPA at 0.76 and the Linear baseline at 0.60, each a mean over five training seeds. When effects were transferred to melanoma co-culture (Frangieh21) every model scored between 0.01 and 0.22 and every printed DEG recall was 0.000; on held-out cell line and cytokine states (Jiang24) scores ranged from 0.17 for Linear to 0.64 for the Decoder baseline, and the paper describes the held-out states in two different ways. In Csendes et al. 2025, whose authors are employees of Turbine Ltd. and built the baselines themselves, predicting the mean training response gave a higher Pearson delta than scGPT and scFoundation, as re-run by the same authors, on all four Perturb-seq datasets: 0.711, 0.557, 0.373 and 0.628 for Adamson, Norman, Replogle K562 and Replogle RPE1, against 0.641, 0.554, 0.327 and 0.596 for scGPT and 0.552, 0.459, 0.269 and 0.471 for scFoundation. Each dataset there has one split and one run per model with no spread printed, so small gaps such as 0.557 against 0.554 cannot be tested. The GEARS developers report a mean squared error of 0.216 for GEARS on Norman, against 0.354 for CPA and 0.551 for a no-change control, with spreads of unstated type. PertEval-scFM, whose authors trained GEARS from scratch, reports an area under its distribution-shift curve of 0.815 for GEARS against 4.484 for an MLP and 4.612 for a mean baseline, lower being better. All of these scores measure agreement with observed expression in held-out perturbations; they do not show which experiments will succeed. (perturbation-ahlmann-20261011-source-data-fig1: sheet 'Panel A', Norman, column l2; perturbation-ahlmann-20261011-source-data-ed-fig2: sheet 'Panel A', Norman, column r2_delta; perturbation-ahlmann-20261011-source-article: Fig. 1 and Extended Data Fig. 2 legends; Methods; lit-cells-networks-20261011-perturbench-v4-source-wu2025: Tables 3, 4 and 7, columns 'Cosine logFC' and 'DEG recall'; perturbation-response-20261009-source-csendes2025-supp: Supplementary Tables 1 and 2; perturbation-response-20261009-source-csendes2025: Results; Methods; Competing interests; coverage-source-gears-supp: Table 6, MSE; perteval-scfm-2025-source: Table 1, Norman single-gene AUSPC)
• genetic_interaction_finding: perturbation-ahlmann-20261011-protocol-norman-double: Genetic interaction finding None of the models was better than the 'no change' baseline at predicting genetic interactions (Fig. 1c), and the same ranking was observed with other metrics (Extended Data Fig. 4). Fig. 1c and Extended Data Fig. 4 were not extracted. (Main text paragraphs 8 and 9)
• Relevance of Adamson: expression prediction for unseen single perturbations (Ahlmann-Eltze et al. 2025 Fig. 2a and Extended Data Fig. 8a) to "Assess models for genetic perturbation experiments": Assessed by Adamson: expression prediction for unseen single perturbations (Ahlmann-Eltze et al. 2025 Fig. 2a and Extended Data Fig. 8a) (perturbation-ahlmann-20261011-source-article: Fig. 2 and Extended Data Fig. 8 legends and Methods; perturbation-ahlmann-20261011-source-data-fig2: sheet "Panel A"; perturbation-ahlmann-20261011-source-data-ed-fig8: sheet "Panel A")
• Relevance of Norman double perturbations: expression prediction for held-out double perturbations (Ahlmann-Eltze et al. 2025 Fig. 1a and Extended Data Fig. 2a) to "Assess models for genetic perturbation experiments": Assessed by Norman double perturbations: expression prediction for held-out double perturbations (Ahlmann-Eltze et al. 2025 Fig. 1a and Extended Data Fig. 2a) (perturbation-ahlmann-20261011-source-article: Fig. 1 and Extended Data Fig. 2 legends and Methods; perturbation-ahlmann-20261011-source-data-fig1: sheet "Panel A"; perturbation-ahlmann-20261011-source-data-ed-fig2: sheet "Panel A")
• Relevance of Replogle K562: expression prediction for unseen single perturbations (Ahlmann-Eltze et al. 2025 Fig. 2a and Extended Data Fig. 8a) to "Assess models for genetic perturbation experiments": Assessed by Replogle K562: expression prediction for unseen single perturbations (Ahlmann-Eltze et al. 2025 Fig. 2a and Extended Data Fig. 8a) (perturbation-ahlmann-20261011-source-article: Fig. 2 and Extended Data Fig. 8 legends and Methods; perturbation-ahlmann-20261011-source-data-fig2: sheet "Panel A"; perturbation-ahlmann-20261011-source-data-ed-fig8: sheet "Panel A")
• Relevance of Replogle RPE1: expression prediction for unseen single perturbations (Ahlmann-Eltze et al. 2025 Fig. 2a and Extended Data Fig. 8a) to "Assess models for genetic perturbation experiments": Assessed by Replogle RPE1: expression prediction for unseen single perturbations (Ahlmann-Eltze et al. 2025 Fig. 2a and Extended Data Fig. 8a) (perturbation-ahlmann-20261011-source-article: Fig. 2 and Extended Data Fig. 8 legends and Methods; perturbation-ahlmann-20261011-source-data-fig2: sheet "Panel A"; perturbation-ahlmann-20261011-source-data-ed-fig8: sheet "Panel A")
• reported_finding: perturbation-ahlmann-20261011-method-linear-model: Reported finding The linear model with gene embeddings from scFoundation or scGPT, or the perturbation embedding from GEARS, performed as well as or better than scGPT and GEARS with their own decoders, and the scFoundation and scGPT gene embeddings outperformed the mean baseline but not consistently the linear model fitted on the training data (Fig. 2c). A linear model with the perturbation embedding pretrained on Replogle data (K562 for Adamson and RPE1, RPE1 for K562) consistently outperformed all other models (paragraph 18 cites no figure for this). Fig. 2c was not extracted. (Main text paragraphs 17 and 18)
• reported_finding: perturbation-ahlmann-20261011-protocol-adamson-single: Reported finding None of the deep learning models was able to consistently outperform the mean prediction or the linear model (Fig. 2a and Extended Data Fig. 8). The paper states this for the single-perturbation benchmark as a whole (Adamson, Replogle K562 and Replogle RPE1), not for each dataset separately. (Main text paragraph 16)
• reported_finding: perturbation-ahlmann-20261011-protocol-norman-double: Reported finding All models had a prediction error substantially higher than the additive baseline (Fig. 1a,b). Other summary statistics, including the Pearson delta and L2 distances over other gene subsets, gave the same overall result (Extended Data Fig. 2). (Main text paragraph 6)
• reported_finding: perturbation-ahlmann-20261011-protocol-replogle-k562-single: Reported finding None of the deep learning models was able to consistently outperform the mean prediction or the linear model (Fig. 2a and Extended Data Fig. 8). The paper states this for the single-perturbation benchmark as a whole (Adamson, Replogle K562 and Replogle RPE1), not for each dataset separately. (Main text paragraph 16)
• reported_finding: perturbation-ahlmann-20261011-protocol-replogle-rpe1-single: Reported finding None of the deep learning models was able to consistently outperform the mean prediction or the linear model (Fig. 2a and Extended Data Fig. 8). The paper states this for the single-perturbation benchmark as a whole (Adamson, Replogle K562 and Replogle RPE1), not for each dataset separately. (Main text paragraph 16)
• scope_note: perturbation-ahlmann-20261011-config-cpa: Scope note CPA was not included in the single-perturbation benchmark, as it is not designed to predict the effects of unseen perturbations. (Source Data Fig. 2 Panel C nevertheless has cpa rows; see the evidence concern on that source.) (Main text paragraph 16)
• scope_note: perturbation-ahlmann-20261011-config-scfoundation: Scope note scFoundation was not included in the single-perturbation benchmark because it required each dataset to exactly match the genes from its own pretraining data, and most of the required genes were missing for the Adamson and Replogle data. (Main text paragraph 16)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.