Relevance of PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits to "Assess models for genetic perturbation experiments"
- Use case
- Assess models for genetic perturbation experiments
- Assessed by
- PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits
- Relevance
- Proxy evidence
- Endpoint
- Area Under the SPECTRA Performance Curve (AUSPC): trapezoidal-rule integral of mean squared error scoring the perturbation-effect delta := P - Xc (Eq. 5, described by the source as a 'log fold change perturbation effect'), across seven increasing-distribution-shift (SPECTRA sparsification) train-test splits; lower is better.
A second, independent, full-text-verified expression-change-prediction comparison including GEARS (trained from scratch by a different group of authors), an MLP baseline (input Xc concatenated with perturbed-gene co-expression features, Eq. 3) and a context-mean baseline. It can inform the same pilot-selection decision as the existing Table 6 comparison; it does not measure experiment-selection yield, and it uses a different dataset preprocessing, split mechanism and metric from Table 6, so the two must not be merged.
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
15 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
Citation locations1 values
| PertEval-scFM (Wenteler et al., ICML 2025), full text perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption. Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Constraints2 values
| PertEval-scFM (Wenteler et al., ICML 2025), full text perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption. Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Endpoint Area Under the SPECTRA Performance Curve (AUSPC): trapezoidal-rule integral of mean squared error scoring the perturbation-effect delta := P - Xc (Eq. 5, described by the source as a 'log fold change perturbation effect'), across seven increasing-distribution-shift (SPECTRA sparsification) train-test splits; lower is better. Context-only references | PertEval-scFM (Wenteler et al., ICML 2025), full text perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption. Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Claimed field links:assessed_by:perteval-scfm-2025-protocol-norman-single-2000hvg-auspc Context-only references | PertEval-scFM (Wenteler et al., ICML 2025), full text perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption. Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Limitations6 values
| PertEval-scFM (Wenteler et al., ICML 2025), full text perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption. Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Pinned records9 values
| PertEval-scFM (Wenteler et al., ICML 2025), full text perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption. Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Rationale A second, independent, full-text-verified expression-change-prediction comparison including GEARS (trained from scratch by a different group of authors), an MLP baseline (input Xc concatenated with perturbed-gene co-expression features, Eq. 3) and a context-mean baseline. It can inform the same pilot-selection decision as the existing Table 6 comparison; it does not measure experiment-selection yield, and it uses a different dataset preprocessing, split mechanism and metric from Table 6, so the two must not be merged. Context-only references | PertEval-scFM (Wenteler et al., ICML 2025), full text perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption. Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Reason Initial applicability review: a second, independently-sourced, full-text-verified comparison of GEARS (trained from scratch) against conventional baselines (an Xc-plus-co-expression MLP baseline, a context-mean baseline) on a differently-preprocessed Norman2019 single-gene subset, with an independent split mechanism (SPECTRA sparsification) and metric (AUSPC), distinct from the existing GEARS Supplementary Table 6 comparison already mapped to this use case. Context-only references | PertEval-scFM (Wenteler et al., ICML 2025), full text perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption. Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Relevance proxy Context-only references | PertEval-scFM (Wenteler et al., ICML 2025), full text perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption. Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Revision 1 Context-only references | PertEval-scFM (Wenteler et al., ICML 2025), full text perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption. Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
Release 2026-11-21-c8bea25e5abb · Record review: source checked
1 source record and release history
- PertEval-scFM (Wenteler et al., ICML 2025), full text · Original source · PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
Technical metadata and extraction receipts
Stable ID: use-case-map-perteval-scfm-norman-single-auspc
- field
- links:assessed_by:perteval-scfm-2025-protocol-norman-single-2000hvg-auspc
- value
- perteval-scfm-2025-protocol-norman-single-2000hvg-auspc
- relevance
- proxy
- endpoint
- Area Under the SPECTRA Performance Curve (AUSPC): trapezoidal-rule integral of mean squared error scoring the perturbation-effect delta := P - Xc (Eq. 5, described by the source as a 'log fold change perturbation effect'), across seven increasing-distribution-shift (SPECTRA sparsification) train-test splits; lower is better.
- rationale
- A second, independent, full-text-verified expression-change-prediction comparison including GEARS (trained from scratch by a different group of authors), an MLP baseline (input Xc concatenated with perturbed-gene co-expression features, Eq. 3) and a context-mean baseline. It can inform the same pilot-selection decision as the existing Table 6 comparison; it does not measure experiment-selection yield, and it uses a different dataset preprocessing, split mechanism and metric from Table 6, so the two must not be merged.
- constraints
- Use the exact PertEval-scFM Table 1 Norman single-gene (2,000 HVG) protocol and configurations; do not combine this evidence with the existing GEARS Supplementary Table 6 comparison, PerturBench, or any other dataset/split/metric in this catalogue.; This intake is limited to the three configurations GEARS, MLP baseline and Mean baseline from Table 1's Norman single-gene section; the five scFM-embedding configurations (Geneformer, scBERT, scFoundation, scGPT, UCE) in the same table, the double-gene section, and the Replogle K562/RPE1 tables in the same paper are not part of this mapping.
- limitations
- AUSPC's printed +/- is the source's own propagated standard error (Appendix F.2, Eqs. F3-F5, propagating per-split triplicate-run MSE uncertainty; main-text Figure 2 caption: 'standard error bars') -- a single, author-reported, propagated quantity, not an independently resampled model-seed standard deviation or confidence interval, and its mathematical derivation is not independently validated here.; Per-sparsification-split (S0.1-S0.7) scored perturbation counts are not printed; the paper's Appendix A.1 gives only the eligible raw dataset perturbation count (105 single-gene perturbations), which is not a scored-per-split count and must not be substituted for one.; Scores the perturbation-effect delta := P - Xc (Eq. 5), not raw post-perturbation expression; the same proxy endpoint class as the existing Table 6 mappings (expression-change prediction, not prospective experiment-selection hit rate) but a different metric construction from Table 6's own Pearson-delta-expression. Does not establish anything about case 6's intervention-selection endpoint.; GEARS here is an independent from-scratch training run by this paper's authors (official implementation, only the train-test split modified, other parameters at default values, no pretrained weights), not the original GEARS authors' own Table 6 checkpoint/run; this is a second, independent GEARS evaluation, not a reproduction of Table 6.; The Mean baseline's input population ('all cells in the same context', per the source's own Section 2.2 text) is not fully specified as control-only or training-only; input budgets differ across the three intaken configurations and are not assumed identical.; This protocol concerns generalisation to unseen perturbations under distribution shift; it does not test or establish forecasting for unseen cells, donors, or cell-line/context transfer.
- revision
- 1
- reason
- Initial applicability review: a second, independently-sourced, full-text-verified comparison of GEARS (trained from scratch) against conventional baselines (an Xc-plus-co-expression MLP baseline, a context-mean baseline) on a differently-preprocessed Norman2019 single-gene subset, with an independent split mechanism (SPECTRA sparsification) and metric (AUSPC), distinct from the existing GEARS Supplementary Table 6 comparison already mapped to this use case.
- citation locators
- source id: perteval-scfm-2025-source; locator: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption.
- source locator
- perteval-scfm-2025-source: Table 1 (Norman single-gene section, rows GEARS / MLP baseline / Mean baseline, AUSPC column, units 10^-2); Appendix A.1 (Norman dataset overview); Section 2.1 (Eq. 3, MLP baseline input); Section 2.2 (Eq. 4 MLP baseline, Eq. 5 perturbation-effect target, GEARS baseline, Mean baseline); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions); main-text Figure 2 caption; Appendix I, Figure I1 caption.
- review
- method: automated-source-review; reviewer: codex; claude-sonnet; reviewer note: Claude Sonnet genetic-perturbation-response evidence-research worker, independently reviewed by Codex; reviewed at: 2026-10-07T13:24:02.149Z; note: Primary-source transcription of three printed table cells (Table 1, Norman single-gene, AUSPC column), independently cross-checked against a freshly re-fetched copy of the same PDF (identical SHA-256), the pinned GitHub code at commit ce48c8b998901c8f8b6275114caac3a4d8543c0b, and the PMLR general publication agreement. No new model execution, independent replication, or qualified human scientific review.
- reviewed evaluations
- perteval-scfm-2025-evaluation-gears; perteval-scfm-2025-evaluation-mean-baseline; perteval-scfm-2025-evaluation-mlp-baseline
- pins
- record id: perteval-scfm-2025-evaluation-gears; fields: status; links; attributes.origin; attributes.comparison; sha256: a9d7ee9238c2f8d58cfa5466545e225d8e37ea41921a413b08f655e916667e16; record id: perteval-scfm-2025-evaluation-mean-baseline; fields: status; links; attributes.origin; attributes.comparison; sha256: f622330f9f03aee7ca632a51e390dfceeb318478fff0de69b34a12a37a54ec3d; record id: perteval-scfm-2025-evaluation-mlp-baseline; fields: status; links; attributes.origin; attributes.comparison; sha256: a0694098cffc69d915c8a19fad9bf6090b7a92f8f311cab8cc75aa3a2bf60e50; record id: perteval-scfm-2025-protocol-norman-single-2000hvg-auspc; fields: status; name; description; facets; source_ids; links; attributes; sha256: 90d4336d06c271a45aba68a318b091861c9442f7441619079efba9c3b3530b64; record id: perteval-scfm-2025-result-gears-auspc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 22482d447ca1808f7f5a61d136b8c2db410a9ae145a4220c4268216544e7fdc3; record id: perteval-scfm-2025-result-mean-baseline-auspc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: fc4c3c8d90de5dbc9075440e4e73d783bc3d487bfdbce35371287daf88d5eff5; record id: perteval-scfm-2025-result-mlp-baseline-auspc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 5095b316ebd95fd91618bd37e1c8257feb6575b05f79e0145c3f311c773bc021; record id: perteval-scfm-2025-source; fields: status; attributes.artifact_sha256; sha256: db327d8402ba69081a8740a322c5f320221b37b694dd0ebddfd05214d0ed78ac; record id: use-case-genetic-perturbation-response; fields: status; name; description; facets; attributes.question; attributes.decision; attributes.inputs; attributes.output; attributes.exclusions; attributes.setting; attributes.clinical_scope; attributes.intended_users; sha256: 440f0f2e30cb7c828212bed728de2420ba8e13521280be58be7bdcdfe57ac973