PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits

Stable ID: perteval-scfm-2025-protocol-norman-single-2000hvg-auspc

areas

cells-spatial-multiomics

tasks

Norman single-gene perturbation effect prediction, 2,000 HVGs, SPECTRA distribution shift

metric

AUSPC (Area Under the SPECTRA Performance Curve of MSE)

metric direction

lower

unit

printed column units: MSE (10^-2) per split; AUSPC (10^-2)

protocol

SPECTRA (Ektefaie et al., 2024) generates seven train-test splits of increasing sparsification probability s=0.1..0.7 (step 0.1), each a controlled-overlap distribution-shift condition for unseen perturbations (Fig. F1) -- this protocol concerns generalisation across perturbations under increasing distribution shift, not a claim about forecasting unseen cells. MSE is computed at each split on the delta=P-Xc perturbation-effect target (Eq. 5), not on raw post-perturbation expression (Table 1 columns S0.1-S0.7); AUSPC is np.trapz(phi, s) over those seven points (Appendix F.2, Eq. F2, Algorithm 1) -- not a ROC-AUC, accuracy or rank-based score. This delta/expression-change MSE is a different metric construction from GEARS Supplementary Table 6's own Pearson-delta-DE metric and must not be conflated with it.

source locator

Table 1 (Norman single-gene section); Appendix A.1 (dataset); Section 2.1 ('Raw expression data', Eq. 3, MLP baseline input); Section 2.2 ('MLP baseline' Eq. 4, 'GEARS baseline' Eq. 5, 'Mean baseline'); Appendix F.2 and Algorithm 1 (AUSPC/uncertainty propagation definitions, Eqs. F2-F5); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits); main-text Figure 2 caption ('standard error bars'); Appendix I, Figure I1 caption (triplicate experiments, '8 train-test splits').

configurations in this intake

GEARS; MLP baseline; Mean baseline

configurations in same table not intaken

Geneformer; scBERT; scFoundation; scGPT; UCE

missing metadata

per split scored sample counts: unreported; see dataset record; random seeds: unreported beyond the word 'triplicate'; exact seed values not printed; hyperparameters: GEARS: official implementation defaults, trained from scratch without pretrained weights (paper's own Methods wording, Section 2.2 region: 'we train GEARS from scratch without using pretrained weights'); MLP baseline / Mean baseline: architecture per Eqs. 4-6 and 'Mean baseline' text in Section 2.2, exact training hyperparameters not separately tabulated for these two baselines in the main text

caveats

This is a different dataset preprocessing (2,000 HVGs, not GEARS Supplementary Table 6's own unstated gene subset), a different split mechanism (SPECTRA sparsification, not Table 6's unstated split) and a different metric construction (AUSPC, a trapezoidal integral of MSE over seven distribution-shift splits, not Table 6's single-split MSE/Pearson-DeltaExpression) from the existing GEARS Supplementary Table 6 catalogue entries for this same use case (protocols gears-2023-supp-table6-task-mse / -task-pearson-de). Do not merge, average or otherwise combine these values with Table 6's.; GEARS in this protocol is trained from scratch by this paper's authors (an independent execution using the official GEARS implementation's default hyperparameters apart from the SPECTRA split), not the original GEARS authors' own reported Table 6 checkpoint/run. This is a second, independent GEARS evaluation, not a reproduction or replication claim for Table 6.; The same Table 1's double-gene section prints a GEARS AUSPC of 0.808 and a printed Mean-baseline-relative delta of 4.254, which is not exactly reproducible from the printed Mean-baseline AUSPC of 4.255 (4.255-0.808=3.447, not 4.254); this inconsistency is specific to the double-gene row, is not resolved or derived here, and is explicitly out of scope for this intake, which covers the single-gene row only.; Table 2 (Replogle RPE1) separately shows a running-text/table-row labeling conflict (text attributes an AUSPC of 0.1251 to the 'Mean baseline', while the table's own row data assigns 0.1251 to Geneformer and 0.1341 to the Mean baseline). This conflict concerns a different dataset (RPE1) not covered by this intake and is noted in the accompanying dossier, not resolved here.; Input populations differ across the three intaken configurations and are not an identical input budget: GEARS uses a graph-based architecture over raw expression and prior-knowledge gene relationships (official implementation); the MLP baseline's input ZGE = Xc (+) Gc concatenates raw control expression with a gene-co-expression matrix for the perturbed gene(s) (Eq. 3), not raw expression alone; the Mean baseline uses no learned input at all (see its own configuration record for the paper's exact, population-ambiguous definition). None of the three should be assumed to have received comparable information.; This protocol concerns generalisation to unseen perturbations under SPECTRA's increasing-sparsification distribution shift. It does not test, and does not establish, forecasting for unseen cells, donors, or cell-line/context transfer.

uncertainty definition

The source describes this quantity as a standard error (main-text Figure 2 caption: 'Average AUSPC (down-arrow) across sparsification probabilities for each model with standard error bars') and separately gives its own propagation formula (Appendix F.2, Eqs. F3-F5): AUSPC's uncertainty is derived from each split's own MSE uncertainty via the trapezoidal integral's partial derivatives (sigma^2 = sum_i (d/2)^2 * sigma_phi_i^2, where d=0.1 is the fixed sparsification step size). These two author statements describe the same quantity and are not in conflict: a propagated quantity can correctly be reported as a standard error. This is recorded as the author's own reported, propagated uncertainty; the F3-F5 derivation is the authors' own formula and its mathematical correctness has not been independently validated here. It must not be read as an independently resampled model-seed standard deviation or a confidence interval. The underlying per-split uncertainty is attributed by the source to triplicate experiments per model (Appendix I, Figure I1 caption: 'Experiments were carried out in triplicate for each model'), not to the main-text Figure 2 region. Figure I1's own caption separately states '8 train-test splits of increasing difficulty' for this same Norman single-gene evaluation, while Table 1 prints seven S-columns (S0.1-S0.7) and Appendix F.2 describes the sparsification probabilities as spanning 0.1 to 0.7. This 7-vs-8 discrepancy between the Figure I1 caption and the Table 1 / F.2 grid is preserved exactly as printed, not resolved; it must not be read as establishing an eighth Table 1 column or a confirmed n_runs=7, and no significance claim is made from any overlapping error bars.

• benchmark: GEARS on PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC
• benchmark: Mean baseline on PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC
• benchmark: MLP baseline on PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC