Evaluation

scFoundation costok GRN on six BEELINE datasets (mean), TFs+1000, non-specific ChIP-seq

Evidence concern: excluded from comparisons

Table 1 labels its middle reference column 'ChIP-seq' and the caption names cell-type-specific ChIP-seq, but its values equal the 'Nonspecific ChIP-seq' mean and std rows of Tables 4 and 6 (for example scGPT costok EPR 4.211±1.703), and Appendix B says cell-type-specific ChIP-seq is excluded from the main comparison. Table 1 is therefore not extracted; Tables 4-7 carry the reference labels.Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 1 caption and column header; Tables 4 and 6, 'Nonspecific ChIP-seq' mean rows; Appendix B text before Table 4

Table 6 is titled 'AUPR' and Table 7 'AUPRC', but both print ratios over a random predictor (values above 1), matching the 'AUPRC ratio' of Table 1; both are recorded as auprc-ratio.Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 6 and Table 7 captions; Table 1 caption

Table 7 (TFs+1000 AUPRC) repeats Table 6 (TFs+500 AUPR) exactly in the GENIE3 column of the STRING, Cell-type-specific ChIP-seq and Nonspecific ChIP-seq blocks: all 18 dataset values and the three mean and std rows are identical (for example STRING hESC 1.6241, mean 3.301, std 2.244). Only the OmniPath block differs. Every other column, and GENIE3 in the EPR Tables 4 and 5, differs between the two gene sets, and the Table 6 values match Table 1. These 21 Table 7 GENIE3 values are likely a copy of the TFs+500 values and should not be compared.Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 7, column group 'ML', column 'GENIE3', blocks 'STRING', 'Cell-type-specific' and 'Nonspecific'; Table 6, same cells; Table 1, GENIE3 AUPRC ratio

Published comparison transcribed from the pinned source; not reproduced.

Evaluation results

1 evaluation · 2 results. Different protocols are not a single leaderboard.

Protocol: Ye et al. 2026 GRN reconstruction, TFs+1000, non-specific ChIP-seq reference · Dataset subset: Six BEELINE datasets, TFs+1000 (Ye et al. 2026)

Sorted by AUPRC ratio (TFs+1000 gene set; non-specific ChIP-seq reference; mean across six datasets) (higher is better). The best value in each column is highlighted. Decimals are rounded for display; each value links to the printed value and its source.

Tested configuration	AUPRC ratio (TFs+1000 gene set; non-specific ChIP-seq reference; mean across six datasets)	Early precision ratio (TFs+1000 gene set; non-specific ChIP-seq reference; mean across six datasets)
scFoundation costok GRN (Ye et al. 2026)Evaluation	1.11	1.19

All 2 result rows with coverage, uncertainty and sources

Filter evaluations

Tested model or methodProtocolDatasetEvaluation setupMetricEvidence origin

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configuration	Protocol and dataset	Finding	Evidence and details
Configuration: scFoundation costok GRN (Ye et al. 2026)	Protocol: Ye et al. 2026 GRN reconstruction, TFs+1000, non-specific ChIP-seq reference Dataset subset: Six BEELINE datasets, TFs+1000 (Ye et al. 2026)	1.11 AUPRC ratio (TFs+1000 gene set; non-specific ChIP-seq reference; mean across six datasets) unitless · higher Uncertainty: SD 0.104 Coverage: Not reported scored / Not reported eligible	Independent external evaluation · Source checked Evidence concern: excluded from comparisons Table 1 labels its middle reference column 'ChIP-seq' and the caption names cell-type-specific ChIP-seq, but its values equal the 'Nonspecific ChIP-seq' mean and std rows of Tables 4 and 6 (for example scGPT costok EPR 4.211±1.703), and Appendix B says cell-type-specific ChIP-seq is excluded from the main comparison. Table 1 is therefore not extracted; Tables 4-7 carry the reference labels.Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 1 caption and column header; Tables 4 and 6, 'Nonspecific ChIP-seq' mean rows; Appendix B text before Table 4 Table 6 is titled 'AUPR' and Table 7 'AUPRC', but both print ratios over a random predictor (values above 1), matching the 'AUPRC ratio' of Table 1; both are recorded as auprc-ratio.Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 6 and Table 7 captions; Table 1 caption Table 7 (TFs+1000 AUPRC) repeats Table 6 (TFs+500 AUPR) exactly in the GENIE3 column of the STRING, Cell-type-specific ChIP-seq and Nonspecific ChIP-seq blocks: all 18 dataset values and the three mean and std rows are identical (for example STRING hESC 1.6241, mean 3.301, std 2.244). Only the OmniPath block differs. Every other column, and GENIE3 in the EPR Tables 4 and 5, differs between the two gene sets, and the Table 6 values match Table 1. These 21 Table 7 GENIE3 values are likely a copy of the TFs+500 values and should not be compared.Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 7, column group 'ML', column 'GENIE3', blocks 'STRING', 'Cell-type-specific' and 'Nonspecific'; Table 6, same cells; Table 1, GENIE3 AUPRC ratio Methods, coverage and source scFoundation costok GRN on six BEELINE datasets (mean), TFs+1000, non-specific ChIP-seq lit-cells-networks-20261011-ye2026grn-protocol-tf-plus-1000-non-specific-chip-seq Aggregation: Not reported Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 7, block 'Nonspecific', row 'mean', column group 'costok', column 'scFoundation'
Configuration: scFoundation costok GRN (Ye et al. 2026)	Protocol: Ye et al. 2026 GRN reconstruction, TFs+1000, non-specific ChIP-seq reference Dataset subset: Six BEELINE datasets, TFs+1000 (Ye et al. 2026)	1.19 Early precision ratio (TFs+1000 gene set; non-specific ChIP-seq reference; mean across six datasets) unitless · higher Uncertainty: SD 0.178 Coverage: Not reported scored / Not reported eligible	Independent external evaluation · Source checked Evidence concern: excluded from comparisons Table 1 labels its middle reference column 'ChIP-seq' and the caption names cell-type-specific ChIP-seq, but its values equal the 'Nonspecific ChIP-seq' mean and std rows of Tables 4 and 6 (for example scGPT costok EPR 4.211±1.703), and Appendix B says cell-type-specific ChIP-seq is excluded from the main comparison. Table 1 is therefore not extracted; Tables 4-7 carry the reference labels.Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 1 caption and column header; Tables 4 and 6, 'Nonspecific ChIP-seq' mean rows; Appendix B text before Table 4 Table 6 is titled 'AUPR' and Table 7 'AUPRC', but both print ratios over a random predictor (values above 1), matching the 'AUPRC ratio' of Table 1; both are recorded as auprc-ratio.Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 6 and Table 7 captions; Table 1 caption Table 7 (TFs+1000 AUPRC) repeats Table 6 (TFs+500 AUPR) exactly in the GENIE3 column of the STRING, Cell-type-specific ChIP-seq and Nonspecific ChIP-seq blocks: all 18 dataset values and the three mean and std rows are identical (for example STRING hESC 1.6241, mean 3.301, std 2.244). Only the OmniPath block differs. Every other column, and GENIE3 in the EPR Tables 4 and 5, differs between the two gene sets, and the Table 6 values match Table 1. These 21 Table 7 GENIE3 values are likely a copy of the TFs+500 values and should not be compared.Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 7, column group 'ML', column 'GENIE3', blocks 'STRING', 'Cell-type-specific' and 'Nonspecific'; Table 6, same cells; Table 1, GENIE3 AUPRC ratio Methods, coverage and source scFoundation costok GRN on six BEELINE datasets (mean), TFs+1000, non-specific ChIP-seq lit-cells-networks-20261011-ye2026grn-protocol-tf-plus-1000-non-specific-chip-seq Aggregation: Not reported Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Table 5, block 'Nonspecific', row 'mean', column group 'costok', column 'scFoundation'

Source checking is not independent reproduction. Release 2026-10-23-5347695500ed.

Research readiness

0 of 4 readiness checks met. These checks assess whether the evidence supports a reproducible investigation; a source-checked score alone does not meet them.

Readiness checks, gaps and artifacts

Release 2026-10-23-5347695500ed · Evidence verified: Not verified

Evidence incomplete

Replay metrics

Exact outcomes, predictions, identifiers and evaluator are connected.

Missing or unresolved evidence

No verified artifact manifest is linked to this exact record.
File checksums match the recorded files: not yet verified
Predictions are matched to the right samples: not yet verified
Score meaning and direction are confirmed: not yet verified
Metrics are recomputed from the saved predictions: not yet verified

Verified: Not verified

Evidence incomplete

Investigate discrepancies

Replay evidence includes annotations and an assessment of dependence. Unknown independence permits descriptive analysis only.

Missing or unresolved evidence

No verified artifact manifest is linked to this exact record.
File checksums match the recorded files: not yet verified
Predictions are matched to the right samples: not yet verified
Score meaning and direction are confirmed: not yet verified
Metrics are recomputed from the saved predictions: not yet verified
Sample annotations are recorded: not yet verified
Dependence between samples is assessed: not yet verified

Verified: Not verified

Evidence incomplete

Run locally

A pinned recipe describes the inputs, environment and resource requirements.

Missing or unresolved evidence

No verified artifact manifest is linked to this exact record.
File checksums match the recorded files: not yet verified
Predictions are matched to the right samples: not yet verified
Score meaning and direction are confirmed: not yet verified
A pinned run recipe exists: not yet verified
Compute requirements are estimated: not yet verified

Verified: Not verified

Evidence incomplete

Validate independently

Separate data and exposure records support an independent test.

Missing or unresolved evidence

No verified artifact manifest is linked to this exact record.
File checksums match the recorded files: not yet verified
Predictions are matched to the right samples: not yet verified
Score meaning and direction are confirmed: not yet verified
Independent validation data exist: not yet verified
Overlap with training data is checked: not yet verified

Verified: Not verified

Readiness describes the evidence in this release. Availability on your computer is checked separately when an investigation runs. Existing data exposure can prevent independent validation even when files are available.

Artifacts and reproduction

No verified artifact manifest is connected to this record yet. The gaps above identify what is needed before analysis can begin.

Evaluation procedure

Ye et al. 2026 GRN reconstruction, TFs+1000, non-specific ChIP-seq reference

Configuration: scFoundation costok GRN (Ye et al. 2026)
Protocol: Ye et al. 2026 GRN reconstruction, TFs+1000, non-specific ChIP-seq reference
Dataset subset: Six BEELINE datasets, TFs+1000 (Ye et al. 2026)

origin: Independent external evaluation
configuration: Primary source as retrieved 2026-10-11
protocol id: lit-cells-networks-20261011-ye2026grn-protocol-tf-plus-1000-non-specific-chip-seq
dataset version: Not reported
split: No train/test split; networks inferred from each full expression matrix
population: Not reported
inputs: BEELINE-preprocessed expression matrix, TFs+1000 gene set (token-embedding configurations use only the gene list)
adaptation: Foundation models zero-shot from pretrained weights; GENIE3, DeepSEM and GRNBoost fitted to each expression matrix
metric implementation: Early precision ratio and AUPRC ratio over a random predictor (Appendix B.2)
aggregation: Mean across the six datasets
budget: Not reported

Metadata review: source checked. Unreported conditions prevent automatic comparisons.

Reproduction

Split: No train/test split; networks inferred from each full expression matrix
Adaptation: Foundation models zero-shot from pretrained weights; GENIE3, DeepSEM and GRNBoost fitted to each expression matrix
Scoring implementation: Early precision ratio and AUPRC ratio over a random predictor (Appendix B.2)

No execution recipe has been verified for this exact configuration and evaluation. A benchmark's general instructions may use different inputs, splits or model settings.

Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

18 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-23-5347695500ed
Property and statement	Original source and location	Review and provenance
Comparison: adaptation Foundation models zero-shot from pretrained weights; GENIE3, DeepSEM and GRNBoost fitted to each expression matrix Context-only references	Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models Original source ↗ Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation' Version: bioRxiv preprint version 1, posted 2026-05-20 Retrieved: 2026-10-11T21:36:31Z	not individually reviewed No individual claim review recorded independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.comparison.adaptation` Source artifact SHA-256: `a6c3aae569b1f1614c71569efb0692f15700aea473c64c7bc0bc660e81330dc7` Hash scope: Exact bytes of one download. bioRxiv serves slightly different PDF bytes per download (41 bytes differed between two downloads 2 minutes apart; pdftotext output identical), so the gzip copy in artifacts/ is the reference. Inspected artifact
Comparison: aggregation Mean across the six datasets Context-only references	Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models Original source ↗ Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation' Version: bioRxiv preprint version 1, posted 2026-05-20 Retrieved: 2026-10-11T21:36:31Z	not individually reviewed No individual claim review recorded independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.comparison.aggregation` Source artifact SHA-256: `a6c3aae569b1f1614c71569efb0692f15700aea473c64c7bc0bc660e81330dc7` Hash scope: Exact bytes of one download. bioRxiv serves slightly different PDF bytes per download (41 bytes differed between two downloads 2 minutes apart; pdftotext output identical), so the gzip copy in artifacts/ is the reference. Inspected artifact
Comparison: budget Not reported Context-only references	Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models Original source ↗ Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation' Version: bioRxiv preprint version 1, posted 2026-05-20 Retrieved: 2026-10-11T21:36:31Z	missing or unspecified No individual claim review recorded independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.comparison.budget` Source artifact SHA-256: `a6c3aae569b1f1614c71569efb0692f15700aea473c64c7bc0bc660e81330dc7` Hash scope: Exact bytes of one download. bioRxiv serves slightly different PDF bytes per download (41 bytes differed between two downloads 2 minutes apart; pdftotext output identical), so the gzip copy in artifacts/ is the reference. Inspected artifact
Comparison: dataset version Not reported Context-only references	Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models Original source ↗ Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation' Version: bioRxiv preprint version 1, posted 2026-05-20 Retrieved: 2026-10-11T21:36:31Z	missing or unspecified No individual claim review recorded independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.comparison.dataset_version` Source artifact SHA-256: `a6c3aae569b1f1614c71569efb0692f15700aea473c64c7bc0bc660e81330dc7` Hash scope: Exact bytes of one download. bioRxiv serves slightly different PDF bytes per download (41 bytes differed between two downloads 2 minutes apart; pdftotext output identical), so the gzip copy in artifacts/ is the reference. Inspected artifact
Comparison: inputs BEELINE-preprocessed expression matrix, TFs+1000 gene set (token-embedding configurations use only the gene list) Context-only references	Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models Original source ↗ Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation' Version: bioRxiv preprint version 1, posted 2026-05-20 Retrieved: 2026-10-11T21:36:31Z	not individually reviewed No individual claim review recorded independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.comparison.inputs` Source artifact SHA-256: `a6c3aae569b1f1614c71569efb0692f15700aea473c64c7bc0bc660e81330dc7` Hash scope: Exact bytes of one download. bioRxiv serves slightly different PDF bytes per download (41 bytes differed between two downloads 2 minutes apart; pdftotext output identical), so the gzip copy in artifacts/ is the reference. Inspected artifact
Comparison: metric implementation Early precision ratio and AUPRC ratio over a random predictor (Appendix B.2) Context-only references	Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models Original source ↗ Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation' Version: bioRxiv preprint version 1, posted 2026-05-20 Retrieved: 2026-10-11T21:36:31Z	not individually reviewed No individual claim review recorded independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.comparison.metric_implementation` Source artifact SHA-256: `a6c3aae569b1f1614c71569efb0692f15700aea473c64c7bc0bc660e81330dc7` Hash scope: Exact bytes of one download. bioRxiv serves slightly different PDF bytes per download (41 bytes differed between two downloads 2 minutes apart; pdftotext output identical), so the gzip copy in artifacts/ is the reference. Inspected artifact
Comparison: population Not reported Context-only references	Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models Original source ↗ Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation' Version: bioRxiv preprint version 1, posted 2026-05-20 Retrieved: 2026-10-11T21:36:31Z	missing or unspecified No individual claim review recorded independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.comparison.population` Source artifact SHA-256: `a6c3aae569b1f1614c71569efb0692f15700aea473c64c7bc0bc660e81330dc7` Hash scope: Exact bytes of one download. bioRxiv serves slightly different PDF bytes per download (41 bytes differed between two downloads 2 minutes apart; pdftotext output identical), so the gzip copy in artifacts/ is the reference. Inspected artifact
Comparison: protocol id lit-cells-networks-20261011-ye2026grn-protocol-tf-plus-1000-non-specific-chip-seq Context-only references	Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models Original source ↗ Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation' Version: bioRxiv preprint version 1, posted 2026-05-20 Retrieved: 2026-10-11T21:36:31Z	not individually reviewed No individual claim review recorded independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.comparison.protocol_id` Source artifact SHA-256: `a6c3aae569b1f1614c71569efb0692f15700aea473c64c7bc0bc660e81330dc7` Hash scope: Exact bytes of one download. bioRxiv serves slightly different PDF bytes per download (41 bytes differed between two downloads 2 minutes apart; pdftotext output identical), so the gzip copy in artifacts/ is the reference. Inspected artifact
Comparison: split No train/test split; networks inferred from each full expression matrix Context-only references	Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models Original source ↗ Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation' Version: bioRxiv preprint version 1, posted 2026-05-20 Retrieved: 2026-10-11T21:36:31Z	not individually reviewed No individual claim review recorded independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.comparison.split` Source artifact SHA-256: `a6c3aae569b1f1614c71569efb0692f15700aea473c64c7bc0bc660e81330dc7` Hash scope: Exact bytes of one download. bioRxiv serves slightly different PDF bytes per download (41 bytes differed between two downloads 2 minutes apart; pdftotext output identical), so the gzip copy in artifacts/ is the reference. Inspected artifact
Origin independent_paper Context-only references	Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models Original source ↗ Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation' Version: bioRxiv preprint version 1, posted 2026-05-20 Retrieved: 2026-10-11T21:36:31Z	not individually reviewed No individual claim review recorded independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.origin` Source artifact SHA-256: `a6c3aae569b1f1614c71569efb0692f15700aea473c64c7bc0bc660e81330dc7` Hash scope: Exact bytes of one download. bioRxiv serves slightly different PDF bytes per download (41 bytes differed between two downloads 2 minutes apart; pdftotext output identical), so the gzip copy in artifacts/ is the reference. Inspected artifact

How to read the evidence

Sources and history

Release 2026-10-23-5347695500ed · Record review: source checked

1 source record and release history

Benchmarking Static Gene Regulatory Network Reconstruction and Dynamic Transition Probing in Single-Cell Foundation Models · Original source · bioRxiv preprint version 1, posted 2026-05-20

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: lit-cells-networks-20261011-ye2026grn-eval-tf-plus-1000-non-specific-chip-seq-six-datasets-scfoundation-costok

areas: biological-networks; single-cell
contexts: research
origin: independent_paper
protocol: lit-cells-networks-20261011-ye2026grn-protocol-tf-plus-1000-non-specific-chip-seq
version: Primary source as retrieved 2026-10-11
comparison: protocol id: lit-cells-networks-20261011-ye2026grn-protocol-tf-plus-1000-non-specific-chip-seq; dataset version: Not reported; split: No train/test split; networks inferred from each full expression matrix; population: Not reported; inputs: BEELINE-preprocessed expression matrix, TFs+1000 gene set (token-embedding configurations use only the gene list); adaptation: Foundation models zero-shot from pretrained weights; GENIE3, DeepSEM and GRNBoost fitted to each expression matrix; metric implementation: Early precision ratio and AUPRC ratio over a random predictor (Appendix B.2); aggregation: Mean across the six datasets; budget: Not reported
source locator: Tables 5 and 7, block 'Nonspecific', row 'mean', column 'costok scFoundation'
missing metadata: comparison.budget: reason: unreported; comparison.dataset version: reason: unreported; comparison.population: reason: unreported

Related records

Suggest a correction