Foundation models against simple controls on unseen perturbations (Csendes et al. 2025)
Proxy evidenceDeveloper-reported: Reported by the developers of the evaluated tool.Independent study: Reported in a study whose authors did not develop the evaluated tool.Pearson delta (higher is better), with the other metrics below each value. The best value in each column is highlighted.
| Tool | Adamson (K562 CRISPRi) | Norman (K562 CRISPRa, singles and pairs) | Replogle K562 (CRISPRi) | Replogle RPE1 (CRISPRi) |
|---|---|---|---|---|
| 0.753 (best in column) | 0.603 | 0.374 | 0.5805 | |
| 0.706 | 0.663 (best in column) | 0.471 | 0.6506 (best in column) | |
| 0.718 | 0.614 | 0.376 | 0.6288 | |
| 0.711 | 0.557 | 0.373 | 0.6284 | |
| 0.641 | 0.554 | 0.327 | 0.5964 | |
| 0.552 | 0.459 | 0.269 | 0.4709 |
Caveats and method
What each column measures:
- Adamson (K562 CRISPRi): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Adamson (K562 CRISPRi)
- Norman (K562 CRISPRa, singles and pairs): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Norman (K562 CRISPRa, singles and pairs)
- Replogle K562 (CRISPRi): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Replogle K562 (CRISPRi)
- Replogle RPE1 (CRISPRi): Pearson correlation of predicted versus observed expression change (Pearson Delta), plus raw-expression and top-20-DE variants, for scGPT, scFoundation, Train Mean and 12 feature-based regressors on unseen perturbations, Replogle RPE1 (CRISPRi)
- Datasets have low perturbation-specific variance (Results, second section), so differences between models are small.
- Top-20 DE metrics include the CRISPR target gene unless stated, which favours scGPT (claim).
- All authors are full-time employees of Turbine Ltd., and one is a founder (Competing interests). They built the Train Mean and regressor baselines, configured and ran the scGPT and scFoundation fine-tuning themselves, and argue that the baselines win; the foundation models were not run by their developers, so their configuration may not be the best available.
- One GEARS perturbation-exclusive split per dataset, within one cell line, with one fine-tuning run per foundation model (best epoch on validation); no seeds, repeats or spread are printed, so small differences between models are not testable.
- The Wilcoxon top-20 DE metric is blank for scGPT and scFoundation in Supplementary Table 2.
- Results paragraphs 7 and 8 cite Supplementary Table 1 for values that are in Supplementary Table 2.
- All perturbations are single-gene CRISPRi in one cell line, so the use case's combination-training and cross-cell-type exclusions are not engaged.
- 79 of the 116 test perturbations are two-gene combinations (9 with neither gene seen in training, 52 with one, 18 with both) and 37 are unseen single genes (Supplementary Table 1); every value pools them. The per-subgroup values (Supplementary Table 3) are not extracted because their metric is not labelled.
- The make-up of the 138 training perturbations is not printed, so whether training included combination examples, which the use case requires for GEARS combination prediction, cannot be confirmed from this source. scFoundation runs through GEARS, so the use case's GEARS combination exclusion applies to it directly.
- The regressors represent a combination as the sum of the two genes' embeddings (Methods 'Baseline models').
Protocols: Adamson et al. 2016 Perturb-seq, K562 CRISPRi single perturbations: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2), Norman et al. 2019 Perturb-seq, K562 CRISPRa single and double perturbations: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2), Replogle et al. 2022 genome-wide Perturb-seq subset, K562 CRISPRi: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2), Replogle et al. 2022 genome-wide Perturb-seq subset, RPE1 CRISPRi: unseen-perturbation expression prediction (Csendes et al. 2025 Supplementary Table 2)