rewirebio.iobenchmarks
Dataset subset

Norman et al. 2019 single-gene perturbations, K562, top 2,000 HVGs (PertEval-scFM Table 1)

Dataset subset reported in PertEval-scFM Table 1, Norman single-gene section. Preprocessing: top 2,000 highly variable genes selected per dataset (scanpy sc.pp.highly_variable_genes); SPECTRA sparsification-probability train-test splits (Section 2.3.2). Exact per-split (S0.1-S0.7) scored perturbation counts are not printed by the source and are not computed here.

Research readiness

These checks assess whether the evidence supports a reproducible investigation. A source-checked score alone does not meet these requirements.

Release 2026-10-07-1448159e6a81 · Evidence verified: Not verified

Evidence incomplete

Replay metrics

Exact outcomes, predictions, identifiers and evaluator are connected.

Missing or unresolved evidence

  • No verified artifact manifest is linked to this exact record.
  • artifact hashes: verification is missing
  • join integrity: verification is missing
  • score semantics: verification is missing
  • metric replay: verification is missing

Verified: Not verified

Evidence incomplete

Investigate discrepancies

Replay evidence includes annotations and an assessment of dependence. Unknown independence permits descriptive analysis only.

Missing or unresolved evidence

  • No verified artifact manifest is linked to this exact record.
  • artifact hashes: verification is missing
  • join integrity: verification is missing
  • score semantics: verification is missing
  • metric replay: verification is missing
  • annotations: verification is missing
  • dependence: verification is missing

Verified: Not verified

Evidence incomplete

Run locally

A pinned recipe describes the inputs, environment and resource requirements.

Missing or unresolved evidence

  • No verified artifact manifest is linked to this exact record.
  • artifact hashes: verification is missing
  • join integrity: verification is missing
  • score semantics: verification is missing
  • recipe pinned: verification is missing
  • resource estimate: verification is missing

Verified: Not verified

Evidence incomplete

Validate independently

Separate data and exposure records support an independent test.

Missing or unresolved evidence

  • No verified artifact manifest is linked to this exact record.
  • artifact hashes: verification is missing
  • join integrity: verification is missing
  • score semantics: verification is missing
  • independent validation: verification is missing
  • overlap checked: verification is missing

Verified: Not verified

Readiness describes the evidence in this release. Availability on your computer is checked separately when an investigation runs. Existing data exposure can prevent independent validation even when files are available.

Artifacts and reproduction

No verified artifact manifest is connected to this record yet. The gaps above identify what is needed before analysis can begin.

Read reviewed discrepancy investigations

Evaluation results

3 evaluations · 3 results. Different protocols are not a single leaderboard.

Filter evaluations

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configurationProtocol and datasetFindingEvidence and details
Configuration: GEARS (trained from scratch, no pretrained weights)Protocol: PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits
Dataset subset: Norman et al. 2019 single-gene perturbations, K562, top 2,000 HVGs (PertEval-scFM Table 1)
0.815 ± 0.039 AUSPC
10^-2 (printed column header units) · lower

Uncertainty: type: author_reported_propagated_standard_error; reported spread: 0.039; note: The source describes this quantity as a standard error (main-text Figure 2 caption: 'Average AUSPC (down-arrow) across sparsification probabilities for each model with standard error bars') and separately gives its own propagation formula (Appendix F.2, Eqs. F3-F5): AUSPC's uncertainty is derived from each split's own MSE uncertainty via the trapezoidal integral's partial derivatives (sigma^2 = sum_i (d/2)^2 * sigma_phi_i^2, where d=0.1 is the fixed sparsification step size). These two author statements describe the same quantity and are not in conflict: a propagated quantity can correctly be reported as a standard error. This is recorded as the author's own reported, propagated uncertainty; the F3-F5 derivation is the authors' own formula and its mathematical correctness has not been independently validated here. It must not be read as an independently resampled model-seed standard deviation or a confidence interval. The underlying per-split uncertainty is attributed by the source to triplicate experiments per model (Appendix I, Figure I1 caption: 'Experiments were carried out in triplicate for each model'), not to the main-text Figure 2 region. Figure I1's own caption separately states '8 train-test splits of increasing difficulty' for this same Norman single-gene evaluation, while Table 1 prints seven S-columns (S0.1-S0.7) and Appendix F.2 describes the sparsification probabilities as spanning 0.1 to 0.7. This 7-vs-8 discrepancy between the Figure I1 caption and the Table 1 / F.2 grid is preserved exactly as printed, not resolved; it must not be read as establishing an eighth Table 1 column or a confirmed n_runs=7, and no significance claim is made from any overlapping error bars.

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

GEARS on PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC

Trapezoidal-rule AUSPC of MSE, scored on the perturbation-effect delta=P-Xc (Eq. 5), across seven SPECTRA sparsification splits (s=0.1..0.7), Norman single-gene, 2,000 HVGs.

Aggregation: Not reported

PertEval-scFM (Wenteler et al., ICML 2025), full text · Table 1, Norman single-gene section, row GEARS, column AUSPC (10^-2).
Configuration: Mean baseline (context mean, no perturbation-specific effect)Protocol: PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits
Dataset subset: Norman et al. 2019 single-gene perturbations, K562, top 2,000 HVGs (PertEval-scFM Table 1)
4.612 ± 0.317 AUSPC
10^-2 (printed column header units) · lower

Uncertainty: type: author_reported_propagated_standard_error; reported spread: 0.317; note: The source describes this quantity as a standard error (main-text Figure 2 caption: 'Average AUSPC (down-arrow) across sparsification probabilities for each model with standard error bars') and separately gives its own propagation formula (Appendix F.2, Eqs. F3-F5): AUSPC's uncertainty is derived from each split's own MSE uncertainty via the trapezoidal integral's partial derivatives (sigma^2 = sum_i (d/2)^2 * sigma_phi_i^2, where d=0.1 is the fixed sparsification step size). These two author statements describe the same quantity and are not in conflict: a propagated quantity can correctly be reported as a standard error. This is recorded as the author's own reported, propagated uncertainty; the F3-F5 derivation is the authors' own formula and its mathematical correctness has not been independently validated here. It must not be read as an independently resampled model-seed standard deviation or a confidence interval. The underlying per-split uncertainty is attributed by the source to triplicate experiments per model (Appendix I, Figure I1 caption: 'Experiments were carried out in triplicate for each model'), not to the main-text Figure 2 region. Figure I1's own caption separately states '8 train-test splits of increasing difficulty' for this same Norman single-gene evaluation, while Table 1 prints seven S-columns (S0.1-S0.7) and Appendix F.2 describes the sparsification probabilities as spanning 0.1 to 0.7. This 7-vs-8 discrepancy between the Figure I1 caption and the Table 1 / F.2 grid is preserved exactly as printed, not resolved; it must not be read as establishing an eighth Table 1 column or a confirmed n_runs=7, and no significance claim is made from any overlapping error bars.

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

Mean baseline on PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC

Trapezoidal-rule AUSPC of MSE, scored on the perturbation-effect delta=P-Xc (Eq. 5), across seven SPECTRA sparsification splits (s=0.1..0.7), Norman single-gene, 2,000 HVGs.

Aggregation: Not reported

PertEval-scFM (Wenteler et al., ICML 2025), full text · Table 1, Norman single-gene section, row Mean baseline, column AUSPC (10^-2).
Configuration: MLP baseline (raw control expression + gene co-expression input, Eq. 3)Protocol: PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC across SPECTRA sparsification splits
Dataset subset: Norman et al. 2019 single-gene perturbations, K562, top 2,000 HVGs (PertEval-scFM Table 1)
4.484 ± 0.299 AUSPC
10^-2 (printed column header units) · lower

Uncertainty: type: author_reported_propagated_standard_error; reported spread: 0.299; note: The source describes this quantity as a standard error (main-text Figure 2 caption: 'Average AUSPC (down-arrow) across sparsification probabilities for each model with standard error bars') and separately gives its own propagation formula (Appendix F.2, Eqs. F3-F5): AUSPC's uncertainty is derived from each split's own MSE uncertainty via the trapezoidal integral's partial derivatives (sigma^2 = sum_i (d/2)^2 * sigma_phi_i^2, where d=0.1 is the fixed sparsification step size). These two author statements describe the same quantity and are not in conflict: a propagated quantity can correctly be reported as a standard error. This is recorded as the author's own reported, propagated uncertainty; the F3-F5 derivation is the authors' own formula and its mathematical correctness has not been independently validated here. It must not be read as an independently resampled model-seed standard deviation or a confidence interval. The underlying per-split uncertainty is attributed by the source to triplicate experiments per model (Appendix I, Figure I1 caption: 'Experiments were carried out in triplicate for each model'), not to the main-text Figure 2 region. Figure I1's own caption separately states '8 train-test splits of increasing difficulty' for this same Norman single-gene evaluation, while Table 1 prints seven S-columns (S0.1-S0.7) and Appendix F.2 describes the sparsification probabilities as spanning 0.1 to 0.7. This 7-vs-8 discrepancy between the Figure I1 caption and the Table 1 / F.2 grid is preserved exactly as printed, not resolved; it must not be read as establishing an eighth Table 1 column or a confirmed n_runs=7, and no significance claim is made from any overlapping error bars.

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

MLP baseline on PertEval-scFM Norman single-gene (2,000 HVGs) AUSPC

Trapezoidal-rule AUSPC of MSE, scored on the perturbation-effect delta=P-Xc (Eq. 5), across seven SPECTRA sparsification splits (s=0.1..0.7), Norman single-gene, 2,000 HVGs.

Aggregation: Not reported

PertEval-scFM (Wenteler et al., ICML 2025), full text · Table 1, Norman single-gene section, row MLP baseline, column AUSPC (10^-2).

Source checking is not independent reproduction. Release 2026-10-07-1448159e6a81.

Subset and evaluation context

This record describes a particular subset or cohort used in an evaluation. Its results do not describe the full dataset.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

8 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-07-1448159e6a81
Property and statementOriginal source and locationReview and provenance
attributes.eligible_raw_cells_total_approx
~110,000 (paper's own Appendix A.1 Table A1; approximate as printed, not a scored count)
Context-only references
PertEval-scFM (Wenteler et al., ICML 2025), full text

Original source ↗

Appendix A.1, Table A1 (dataset overview); Table 1 (Norman single-gene section); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits)

Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
Retrieved: 2026-10-07T11:39:51Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.eligible_raw_cells_total_approx

Source artifact SHA-256: c116a153872645d5c91b9ee836df3945cd8ffd02a6e736f58f854c4b70978bcc

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

attributes.eligible_raw_control_cells_approx
~12,000 (paper's own Appendix A.1 Table A1; approximate as printed, not a scored count)
Context-only references
PertEval-scFM (Wenteler et al., ICML 2025), full text

Original source ↗

Appendix A.1, Table A1 (dataset overview); Table 1 (Norman single-gene section); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits)

Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
Retrieved: 2026-10-07T11:39:51Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.eligible_raw_control_cells_approx

Source artifact SHA-256: c116a153872645d5c91b9ee836df3945cd8ffd02a6e736f58f854c4b70978bcc

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

attributes.eligible_raw_perturbations_double_gene_not_in_this_intake
91
Context-only references
PertEval-scFM (Wenteler et al., ICML 2025), full text

Original source ↗

Appendix A.1, Table A1 (dataset overview); Table 1 (Norman single-gene section); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits)

Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
Retrieved: 2026-10-07T11:39:51Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.eligible_raw_perturbations_double_gene_not_in_this_intake

Source artifact SHA-256: c116a153872645d5c91b9ee836df3945cd8ffd02a6e736f58f854c4b70978bcc

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

attributes.eligible_raw_perturbations_single_gene
105
Context-only references
PertEval-scFM (Wenteler et al., ICML 2025), full text

Original source ↗

Appendix A.1, Table A1 (dataset overview); Table 1 (Norman single-gene section); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits)

Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
Retrieved: 2026-10-07T11:39:51Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.eligible_raw_perturbations_single_gene

Source artifact SHA-256: c116a153872645d5c91b9ee836df3945cd8ffd02a6e736f58f854c4b70978bcc

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

attributes.source_dataset
Norman et al. 2019 (K562, CRISPRa Perturb-seq)
Context-only references
PertEval-scFM (Wenteler et al., ICML 2025), full text

Original source ↗

Appendix A.1, Table A1 (dataset overview); Table 1 (Norman single-gene section); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits)

Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
Retrieved: 2026-10-07T11:39:51Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.source_dataset

Source artifact SHA-256: c116a153872645d5c91b9ee836df3945cd8ffd02a6e736f58f854c4b70978bcc

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

attributes.source_locator
Appendix A.1, Table A1 (dataset overview); Table 1 (Norman single-gene section); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits)
Context-only references
PertEval-scFM (Wenteler et al., ICML 2025), full text

Original source ↗

Appendix A.1, Table A1 (dataset overview); Table 1 (Norman single-gene section); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits)

Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
Retrieved: 2026-10-07T11:39:51Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.source_locator

Source artifact SHA-256: c116a153872645d5c91b9ee836df3945cd8ffd02a6e736f58f854c4b70978bcc

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

description
Dataset subset reported in PertEval-scFM Table 1, Norman single-gene section. Preprocessing: top 2,000 highly variable genes selected per dataset (scanpy sc.pp.highly_variable_genes); SPECTRA sparsification-probability train-test splits (Section 2.3.2). Exact per-split (S0.1-S0.7) scored perturbation counts are not printed by the source and are not computed here.
Context-only references
PertEval-scFM (Wenteler et al., ICML 2025), full text

Original source ↗

Appendix A.1, Table A1 (dataset overview); Table 1 (Norman single-gene section); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits)

Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
Retrieved: 2026-10-07T11:39:51Z

not individually reviewed

No individual claim review recorded

Audit details

Field: description

Source artifact SHA-256: c116a153872645d5c91b9ee836df3945cd8ffd02a6e736f58f854c4b70978bcc

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

name
Norman et al. 2019 single-gene perturbations, K562, top 2,000 HVGs (PertEval-scFM Table 1)
Context-only references
PertEval-scFM (Wenteler et al., ICML 2025), full text

Original source ↗

Appendix A.1, Table A1 (dataset overview); Table 1 (Norman single-gene section); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits)

Version: PMLR v267 wenteler25a (as served by the PMLR-affiliated mlresearch/v267 GitHub mirror; ETag "ed9f0fe44cf6edc939ee6950d024f65dcd8dd6f16414bd9f5297cda9395d6e58" at retrieval)
Retrieved: 2026-10-07T11:39:51Z

not individually reviewed

No individual claim review recorded

Audit details

Field: name

Source artifact SHA-256: c116a153872645d5c91b9ee836df3945cd8ffd02a6e736f58f854c4b70978bcc

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Sources and history

View linked audit checks and correction history

Release 2026-10-07-1448159e6a81 · Record review: source checked

1 source records and release historyDownload this release
Technical metadata and extraction receipts

Stable ID: perteval-scfm-2025-dataset-norman-single-2000hvg

areas
cells-spatial-multiomics
source dataset
Norman et al. 2019 (K562, CRISPRa Perturb-seq)
source locator
Appendix A.1, Table A1 (dataset overview); Table 1 (Norman single-gene section); Section 2.1.1 (HVG selection); Section 2.3.2 (SPECTRA splits)
eligible raw perturbations single gene
105
eligible raw perturbations double gene not in this intake
91
eligible raw cells total approx
~110,000 (paper's own Appendix A.1 Table A1; approximate as printed, not a scored count)
eligible raw control cells approx
~12,000 (paper's own Appendix A.1 Table A1; approximate as printed, not a scored count)
missing metadata
per split scored perturbation count: unreported for each of S0.1-S0.7; the eligible raw single-gene count (105) above is the compiled dataset total, not a per-split scored count, and must not be substituted for one
Related records

Suggest a correction