PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis (arXiv v4)

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 3 configurations
• 7 protocols
• 5 dataset subsets
• 57 evaluations
• 340 results
• 4 evidence claims

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• Evidence summary: genetic perturbation response prediction: Summary The five sources whose comparisons are shown on this page score how well models predict expression changes after genetic perturbations they were not trained on. Each uses its own split, gene set and metric, so their values are not pooled. Ahlmann-Eltze et al. 2025, who did not develop the deep models and ran them all themselves, scored held-out two-gene CRISPRa perturbations in K562 cells (Norman), training on every single-gene perturbation and half of the pairs. A baseline that adds the two single-gene effects had a mean L2 distance of 2.766 and a Pearson delta of 0.892 over the 1,000 most highly expressed genes, against 3.478 and 0.773 for Geneformer with a linear decoder, 4.096 and 0.631 for GEARS, 4.279 and 0.610 for both UCE and scBERT, 4.616 and 0.527 for scGPT, 4.722 and 0.537 for scFoundation, and 12.205 and 0.146 for CPA; predicting no change gave an L2 distance of 5.616. Rewire computed these means from the authors' Source Data, no spread is stated, and the UCE and scBERT runs used ground-truth expression of the perturbed genes, a possible test leakage. In PerturBench v4, a preprint whose authors trained every model and built the Linear, Latent Additive and Decoder baselines, the cosine similarity of predicted log fold change on held-out Norman pairs ranged from 0.41 for Biolord and 0.44 for GEARS to 0.79 for the Latent Additive baseline, with CPA at 0.76 and the Linear baseline at 0.60, each a mean over five training seeds. When effects were transferred to melanoma co-culture (Frangieh21) every model scored between 0.01 and 0.22 and every printed DEG recall was 0.000; on held-out cell line and cytokine states (Jiang24) scores ranged from 0.17 for Linear to 0.64 for the Decoder baseline, and the paper describes the held-out states in two different ways. In Csendes et al. 2025, whose authors are employees of Turbine Ltd. and built the baselines themselves, predicting the mean training response gave a higher Pearson delta than scGPT and scFoundation, as re-run by the same authors, on all four Perturb-seq datasets: 0.711, 0.557, 0.373 and 0.628 for Adamson, Norman, Replogle K562 and Replogle RPE1, against 0.641, 0.554, 0.327 and 0.596 for scGPT and 0.552, 0.459, 0.269 and 0.471 for scFoundation. Each dataset there has one split and one run per model with no spread printed, so small gaps such as 0.557 against 0.554 cannot be tested. The GEARS developers report a mean squared error of 0.216 for GEARS on Norman, against 0.354 for CPA and 0.551 for a no-change control, with spreads of unstated type. PertEval-scFM, whose authors trained GEARS from scratch, reports an area under its distribution-shift curve of 0.815 for GEARS against 4.484 for an MLP and 4.612 for a mean baseline, lower being better. All of these scores measure agreement with observed expression in held-out perturbations; they do not show which experiments will succeed. (perturbation-ahlmann-20261011-source-data-fig1: sheet 'Panel A', Norman, column l2; perturbation-ahlmann-20261011-source-data-ed-fig2: sheet 'Panel A', Norman, column r2_delta; perturbation-ahlmann-20261011-source-article: Fig. 1 and Extended Data Fig. 2 legends; Methods; lit-cells-networks-20261011-perturbench-v4-source-wu2025: Tables 3, 4 and 7, columns 'Cosine logFC' and 'DEG recall'; perturbation-response-20261009-source-csendes2025-supp: Supplementary Tables 1 and 2; perturbation-response-20261009-source-csendes2025: Results; Methods; Competing interests; coverage-source-gears-supp: Table 6, MSE; perteval-scfm-2025-source: Table 1, Norman single-gene AUSPC)
• Relevance of Frangieh21 co-culture transfer (genetic perturbations) to "Assess models for genetic perturbation experiments": Assessed by PerturBench v4 covariate transfer from mono-culture to co-culture on Frangieh21 (lit-cells-networks-20261011-perturbench-v4-source-wu2025: Tables 7 and 8; Appendix D.4 'Frangieh21 splits')
• Relevance of Jiang24 held-out states (genetic perturbations) to "Assess models for genetic perturbation experiments": Assessed by PerturBench v4 covariate transfer to held-out perturbations in held-out biological states on Jiang24 (lit-cells-networks-20261011-perturbench-v4-source-wu2025: Tables 4 and 11; Appendix D.4 'Jiang24 splits')
• Relevance of Norman19 dual perturbations (K562 CRISPRa) to "Assess models for genetic perturbation experiments": Assessed by PerturBench v4 combination prediction of held-out dual perturbations on Norman19 (lit-cells-networks-20261011-perturbench-v4-source-wu2025: Tables 3 and 6; Section 4.2; Appendix D.4-D.5)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.