PertEval-scFM (Wenteler et al., ICML 2025), full text
Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR vol. 267, pp. 66633-66677 · PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
What this source contributed
Records in this release that cite this source. A record can cite several sources.
Results
Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.
Claims
Statements this source supports, each with where in the source it was checked.
- Evidence summary: genetic perturbation response prediction: Summary The five sources whose comparisons are shown on this page score how well models predict expression changes after genetic perturbations they were not trained on. Each uses its own split, gene set and metric, so their values are not pooled. Ahlmann-Eltze et al. 2025, who did not develop the deep models and ran them all themselves, scored held-out two-gene CRISPRa perturbations in K562 cells (Norman), training on every single-gene perturbation and half of the pairs. A baseline that adds the two single-gene effects had a mean L2 distance of 2.766 and a Pearson delta of 0.892 over the 1,000 most highly expressed genes, against 3.478 and 0.773 for Geneformer with a linear decoder, 4.096 and 0.631 for GEARS, 4.279 and 0.610 for both UCE and scBERT, 4.616 and 0.527 for scGPT, 4.722 and 0.537 for scFoundation, and 12.205 and 0.146 for CPA; predicting no change gave an L2 distance of 5.616. Rewire computed these means from the authors' Source Data, no spread is stated, and the UCE and scBERT runs used ground-truth expression of the perturbed genes, a possible test leakage. In PerturBench v4, a preprint whose authors trained every model and built the Linear, Latent Additive and Decoder baselines, the cosine similarity of predicted log fold change on held-out Norman pairs ranged from 0.41 for Biolord and 0.44 for GEARS to 0.79 for the Latent Additive baseline, with CPA at 0.76 and the Linear baseline at 0.60, each a mean over five training seeds. When effects were transferred to melanoma co-culture (Frangieh21) every model scored between 0.01 and 0.22 and every printed DEG recall was 0.000; on held-out cell line and cytokine states (Jiang24) scores ranged from 0.17 for Linear to 0.64 for the Decoder baseline, and the paper describes the held-out states in two different ways. In Csendes et al. 2025, whose authors are employees of Turbine Ltd. and built the baselines themselves, predicting the mean training response gave a higher Pearson delta than scGPT and scFoundation, as re-run by the same authors, on all four Perturb-seq datasets: 0.711, 0.557, 0.373 and 0.628 for Adamson, Norman, Replogle K562 and Replogle RPE1, against 0.641, 0.554, 0.327 and 0.596 for scGPT and 0.552, 0.459, 0.269 and 0.471 for scFoundation. Each dataset there has one split and one run per model with no spread printed, so small gaps such as 0.557 against 0.554 cannot be tested. The GEARS developers report a mean squared error of 0.216 for GEARS on Norman, against 0.354 for CPA and 0.551 for a no-change control, with spreads of unstated type. PertEval-scFM, whose authors trained GEARS from scratch, reports an area under its distribution-shift curve of 0.815 for GEARS against 4.484 for an MLP and 4.612 for a mean baseline, lower being better. All of these scores measure agreement with observed expression in held-out perturbations; they do not show which experiments will succeed. (perturbation-ahlmann-20261011-source-data-fig1: sheet 'Panel A', Norman, column l2; perturbation-ahlmann-20261011-source-data-ed-fig2: sheet 'Panel A', Norman, column r2_delta; perturbation-ahlmann-20261011-source-article: Fig. 1 and Extended Data Fig. 2 legends; Methods; lit-cells-networks-20261011-perturbench-v4-source-wu2025: Tables 3, 4 and 7, columns 'Cosine logFC' and 'DEG recall'; perturbation-response-20261009-source-csendes2025-supp: Supplementary Tables 1 and 2; perturbation-response-20261009-source-csendes2025: Results; Methods; Competing interests; coverage-source-gears-supp: Table 6, MSE; perteval-scfm-2025-source: Table 1, Norman single-gene AUSPC)
- Relevance of PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits to "Assess models for genetic perturbation experiments": Assessed by PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits (perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption.)
Models, benchmarks and other records
Configurations (3)
Protocols (1)
Dataset subsets (1)
Evaluations (3)
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
16 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Article licence PMLR's general Publication Agreement (https://proceedings.mlr.press/pmlr-license-agreement.html, independently read 2026-10-07T12:22:03Z) requires every PMLR author to grant "a Creative Commons Attribution 4.0 International License" in the article, with an attribution requirement of "a citation to the original publication of the article in the proceedings as well as a hyperlink to the PMLR web site linking to the original paper." This general agreement, not an individual-page badge, is the basis for treating this article as CC BY 4.0: the individual article landing page itself shows no Creative Commons badge, only the separate "Copyright (c) The authors and PMLR 2026" footer line (both facts recorded, neither substituted for the other). Source metadata | PertEval-scFM (Wenteler et al., ICML 2025), full text No field-specific location recorded Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Artifact SHA-256 c116a153872645d5c91b9ee836df3945cd8ffd02a6e736f58f854c4b70978bcc Source metadata | PertEval-scFM (Wenteler et al., ICML 2025), full text No field-specific location recorded Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Artifact url https://raw.githubusercontent.com/mlresearch/v267/main/assets/wenteler25a/wenteler25a.pdf Source metadata | PertEval-scFM (Wenteler et al., ICML 2025), full text No field-specific location recorded Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Code licence MIT, confirmed via the GitHub repository API license endpoint on 2026-10-07, and independently re-confirmed by reading the pinned LICENSE blob at main commit ce48c8b998901c8f8b6275114caac3a4d8543c0b (blob 8f7d6532772dcd06951192fe441e00d02e0848c2). This licenses the code repository only: it does not license the paper's datasets, any other model's pretrained weights, and it does not identify which code revision produced the published Table 1/2 run (that mapping is unverified). Source metadata | PertEval-scFM (Wenteler et al., ICML 2025), full text No field-specific location recorded Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Code url https://github.com/aaronwtr/PertEval Source metadata | PertEval-scFM (Wenteler et al., ICML 2025), full text No field-specific location recorded Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Retrieved at 2026-10-07T11:39:51Z Source metadata | PertEval-scFM (Wenteler et al., ICML 2025), full text No field-specific location recorded Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Url https://proceedings.mlr.press/v267/wenteler25a.html Source metadata | PertEval-scFM (Wenteler et al., ICML 2025), full text No field-specific location recorded Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Venue Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR vol. 267, pp. 66633-66677 Source metadata | PertEval-scFM (Wenteler et al., ICML 2025), full text No field-specific location recorded Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Version PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) Source metadata | PertEval-scFM (Wenteler et al., ICML 2025), full text No field-specific location recorded Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Description Pinned primary PDF. Review is limited to Table 1 (Norman single-gene, 2,000 HVGs) GEARS / MLP baseline / Mean baseline AUSPC rows. The five scFM-embedding rows in the same table and the double-gene/Replogle K562/Replogle RPE1 tables are not reviewed for intake by this source record. Source metadata | PertEval-scFM (Wenteler et al., ICML 2025), full text No field-specific location recorded Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
Release 2026-11-21-c8bea25e5abb · Record review: source checked
Pinned primary PDF. Review is limited to Table 1 (Norman single-gene, 2,000 HVGs) GEARS / MLP baseline / Mean baseline AUSPC rows. The five scFM-embedding rows in the same table and the double-gene/Replogle K562/Replogle RPE1 tables are not reviewed for intake by this source record.
Technical metadata and extraction receipts
Stable ID: perteval-scfm-2025-source
- areas
- single-cell
- url
- https://proceedings.mlr.press/v267/wenteler25a.html
- artifact url
- https://raw.githubusercontent.com/mlresearch/v267/main/assets/wenteler25a/wenteler25a.pdf
- artifact sha256
- c116a153872645d5c91b9ee836df3945cd8ffd02a6e736f58f854c4b70978bcc
- retrieved at
- 2026-10-07T11:39:51Z
- venue
- Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR vol. 267, pp. 66633-66677
- version
- PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
- code url
- https://github.com/aaronwtr/PertEval
- article licence
- PMLR's general Publication Agreement (https://proceedings.mlr.press/pmlr-license-agreement.html, independently read 2026-10-07T12:22:03Z) requires every PMLR author to grant "a Creative Commons Attribution 4.0 International License" in the article, with an attribution requirement of "a citation to the original publication of the article in the proceedings as well as a hyperlink to the PMLR web site linking to the original paper." This general agreement, not an individual-page badge, is the basis for treating this article as CC BY 4.0: the individual article landing page itself shows no Creative Commons badge, only the separate "Copyright (c) The authors and PMLR 2026" footer line (both facts recorded, neither substituted for the other).
- code licence
- MIT, confirmed via the GitHub repository API license endpoint on 2026-10-07, and independently re-confirmed by reading the pinned LICENSE blob at main commit ce48c8b998901c8f8b6275114caac3a4d8543c0b (blob 8f7d6532772dcd06951192fe441e00d02e0848c2). This licenses the code repository only: it does not license the paper's datasets, any other model's pretrained weights, and it does not identify which code revision produced the published Table 1/2 run (that mapping is unverified).