rewirebio.iobenchmarks
Source

Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19) · Preprint

DOI: 10.48550/arXiv.2607.17227Original source

What this source contributed

Records in this release that cite this source. A record can cite several sources.

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Model or configurationMetricValueDatasetEvaluation
CellPLM (continual, Chen et al. 2026)Adjusted rand index0.84 unitlessATAA (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, ATAA
CellPLM (continual, Chen et al. 2026)Normalized mutual information0.82 unitlessATAA (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, ATAA
CellPLM (continual, Chen et al. 2026)Silhouette score (against reference labels on the model embedding; range -1 to 1; distance stated as cosine in some model sections of the Methods)0.26 unitlessATAA (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, ATAA
CellPLM (continual, Chen et al. 2026)Adjusted rand index0.16 unitlessBC (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, BC
CellPLM (continual, Chen et al. 2026)Normalized mutual information0.31 unitlessBC (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, BC
CellPLM (continual, Chen et al. 2026)Silhouette score (against reference labels on the model embedding; range -1 to 1; distance stated as cosine in some model sections of the Methods)0.10 unitlessBC (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, BC
CellPLM (continual, Chen et al. 2026)Adjusted rand index0.28 unitlessBMMC (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, BMMC
CellPLM (continual, Chen et al. 2026)Normalized mutual information0.66 unitlessBMMC (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, BMMC
CellPLM (continual, Chen et al. 2026)Silhouette score (against reference labels on the model embedding; range -1 to 1; distance stated as cosine in some model sections of the Methods)0.12 unitlessBMMC (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, BMMC
CellPLM (continual, Chen et al. 2026)Adjusted rand index0.37 unitlessHS (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, HS
CellPLM (continual, Chen et al. 2026)Normalized mutual information0.52 unitlessHS (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, HS
CellPLM (continual, Chen et al. 2026)Silhouette score (against reference labels on the model embedding; range -1 to 1; distance stated as cosine in some model sections of the Methods)0.14 unitlessHS (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, HS
CellPLM (continual, Chen et al. 2026)Adjusted rand index0.27 unitlessME (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, ME
CellPLM (continual, Chen et al. 2026)Normalized mutual information0.40 unitlessME (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, ME
CellPLM (continual, Chen et al. 2026)Silhouette score (against reference labels on the model embedding; range -1 to 1; distance stated as cosine in some model sections of the Methods)0.09 unitlessME (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, ME
CellPLM (continual, Chen et al. 2026)Adjusted rand index0.15 unitlessMH (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, MH
CellPLM (continual, Chen et al. 2026)Normalized mutual information0.21 unitlessMH (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, MH
CellPLM (continual, Chen et al. 2026)Silhouette score (against reference labels on the model embedding; range -1 to 1; distance stated as cosine in some model sections of the Methods)0.06 unitlessMH (Chen et al. 2026)CellPLM on Chen et al. 2026 continually pretrained clustering, MH
CellPLM (gears-decoder, Chen et al. 2026)MSE (all genes, predicted versus observed post-perturbation expression)0.0091 unitlessAdamson (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Adamson
CellPLM (gears-decoder, Chen et al. 2026)MSE (top 20 differentially expressed genes per perturbation condition, predicted versus observed post-perturbation expression)0.2778 unitlessAdamson (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Adamson
CellPLM (gears-decoder, Chen et al. 2026)Pearson (all genes, predicted versus observed post-perturbation expression)0.9876 unitlessAdamson (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Adamson
CellPLM (gears-decoder, Chen et al. 2026)Pearson (top 20 differentially expressed genes per perturbation condition, predicted versus observed post-perturbation expression)0.9236 unitlessAdamson (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Adamson
CellPLM (gears-decoder, Chen et al. 2026)Pearson delta (all genes in the intersected gene set)0.6487 unitlessAdamson (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Adamson
CellPLM (gears-decoder, Chen et al. 2026)Pearson delta (top 20 differentially expressed genes per perturbation condition)0.7112 unitlessAdamson (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Adamson
CellPLM (gears-decoder, Chen et al. 2026)MSE (all genes, predicted versus observed post-perturbation expression)0.0064 unitlessNorman (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Norman
CellPLM (gears-decoder, Chen et al. 2026)MSE (top 20 differentially expressed genes per perturbation condition, predicted versus observed post-perturbation expression)0.4034 unitlessNorman (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Norman
CellPLM (gears-decoder, Chen et al. 2026)Pearson (all genes, predicted versus observed post-perturbation expression)0.9826 unitlessNorman (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Norman
CellPLM (gears-decoder, Chen et al. 2026)Pearson (top 20 differentially expressed genes per perturbation condition, predicted versus observed post-perturbation expression)0.8724 unitlessNorman (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Norman
CellPLM (gears-decoder, Chen et al. 2026)Pearson delta (all genes in the intersected gene set)0.4727 unitlessNorman (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Norman
CellPLM (gears-decoder, Chen et al. 2026)Pearson delta (top 20 differentially expressed genes per perturbation condition)0.3982 unitlessNorman (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Norman
CellPLM (gears-decoder, Chen et al. 2026)MSE (all genes, predicted versus observed post-perturbation expression)0.0068 unitlessReplogle (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Replogle
CellPLM (gears-decoder, Chen et al. 2026)MSE (top 20 differentially expressed genes per perturbation condition, predicted versus observed post-perturbation expression)0.1844 unitlessReplogle (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Replogle
CellPLM (gears-decoder, Chen et al. 2026)Pearson (all genes, predicted versus observed post-perturbation expression)0.9868 unitlessReplogle (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Replogle
CellPLM (gears-decoder, Chen et al. 2026)Pearson (top 20 differentially expressed genes per perturbation condition, predicted versus observed post-perturbation expression)0.9406 unitlessReplogle (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Replogle
CellPLM (gears-decoder, Chen et al. 2026)Pearson delta (all genes in the intersected gene set)0.3801 unitlessReplogle (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Replogle
CellPLM (gears-decoder, Chen et al. 2026)Pearson delta (top 20 differentially expressed genes per perturbation condition)0.4604 unitlessReplogle (Chen et al. 2026)CellPLM on Chen et al. 2026 genetic perturbation prediction, Replogle
CellPLM (fine-tuned, Chen et al. 2026)Accuracy (held-out test set)0.97 unitlessATAA (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, ATAA
CellPLM (fine-tuned, Chen et al. 2026)AUPRC (held-out test set; macro-averaged over classes)0.98 unitlessATAA (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, ATAA
CellPLM (fine-tuned, Chen et al. 2026)AUROC (held-out test set; macro-averaged over classes)0.99 unitlessATAA (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, ATAA
CellPLM (fine-tuned, Chen et al. 2026)Macro F1 (held-out test set)0.95 unitlessATAA (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, ATAA
CellPLM (fine-tuned, Chen et al. 2026)Precision (held-out test set; described as global in the Methods, averaging not further defined)0.97 unitlessATAA (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, ATAA
CellPLM (fine-tuned, Chen et al. 2026)Recall (held-out test set; described as global in the Methods, averaging not further defined)0.97 unitlessATAA (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, ATAA
CellPLM (fine-tuned, Chen et al. 2026)Accuracy (held-out test set)0.86 unitlessBMMC (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, BMMC
CellPLM (fine-tuned, Chen et al. 2026)AUPRC (held-out test set; macro-averaged over classes)0.80 unitlessBMMC (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, BMMC
CellPLM (fine-tuned, Chen et al. 2026)AUROC (held-out test set; macro-averaged over classes)0.99 unitlessBMMC (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, BMMC
CellPLM (fine-tuned, Chen et al. 2026)Macro F1 (held-out test set)0.73 unitlessBMMC (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, BMMC
CellPLM (fine-tuned, Chen et al. 2026)Precision (held-out test set; described as global in the Methods, averaging not further defined)0.86 unitlessBMMC (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, BMMC
CellPLM (fine-tuned, Chen et al. 2026)Recall (held-out test set; described as global in the Methods, averaging not further defined)0.86 unitlessBMMC (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, BMMC
CellPLM (fine-tuned, Chen et al. 2026)Accuracy (held-out test set)0.93 unitlessHS (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, HS
CellPLM (fine-tuned, Chen et al. 2026)AUPRC (held-out test set; macro-averaged over classes)0.96 unitlessHS (Chen et al. 2026)CellPLM on Chen et al. 2026 supervised cell type annotation, HS

Claims

Statements this source supports, each with where in the source it was checked.

Models, benchmarks and other records

Models (6)

Configurations (21)

Benchmarks (1)

Protocols (25)

Dataset subsets (10)

Evaluations (93)

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

24 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-23-5347695500ed
Property and statementOriginal source and locationReview and provenance
Artifact SHA-256
8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da
Source metadata
Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Original source ↗

No field-specific location recorded

Version: arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
Retrieved: 2026-10-11T21:29:34Z

catalogued

No individual claim review recorded

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.artifact_sha256

Source artifact SHA-256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Artifact url
https://arxiv.org/pdf/2607.17227v2
Source metadata
Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Original source ↗

No field-specific location recorded

Version: arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
Retrieved: 2026-10-11T21:29:34Z

catalogued

No individual claim review recorded

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.artifact_url

Source artifact SHA-256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Doi
10.48550/arXiv.2607.17227
Source metadata
Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Original source ↗

No field-specific location recorded

Version: arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
Retrieved: 2026-10-11T21:29:34Z

catalogued

No individual claim review recorded

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.doi

Source artifact SHA-256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Evidence concerns
1 values
  • source id
    lit-cells-networks-20261011-chen2026-source-chen2026
    message
    Table 4 precision and recall equal accuracy in all eight Nicheformer and CellPLM rows, as micro (global) averaging would give, but recall differs from accuracy in every scGPT-spatial, GenePT-w, scELMo and Novae row. The Methods call precision and recall 'global' without defining the averaging, so they may be averaged differently between models and should not be compared across models.
    source locator
    Table 4, columns 'Accuracy', 'Precision' and 'Recall'; Methods 'Cell type annotation' (page 25)
    artifact sha256
    8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da
    review date
    2026-10-12
Source metadata
Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Original source ↗

No field-specific location recorded

Version: arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
Retrieved: 2026-10-11T21:29:34Z

catalogued

No individual claim review recorded

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.evidence_concerns

Source artifact SHA-256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Licence
CC-BY-SA-4.0
Source metadata
Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Original source ↗

No field-specific location recorded

Version: arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
Retrieved: 2026-10-11T21:29:34Z

catalogued

No individual claim review recorded

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.licence

Source artifact SHA-256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Limitations
3 values
  • Preprint, not peer reviewed.
  • Single run per cell with a fixed seed (42); no spread is printed.
  • Spatial dataset codes BC, MH and ME are not defined in the tables; the Figure 2 legend defines them as BC/HBCA1 (human breast cancer), MH/MHPC (mouse hippocampus) and ME/MOSTA E9.5 (mouse embryo).
Source metadata
Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Original source ↗

No field-specific location recorded

Version: arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
Retrieved: 2026-10-11T21:29:34Z

catalogued

No individual claim review recorded

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.limitations

Source artifact SHA-256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Media type
application/pdf
Source metadata
Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Original source ↗

No field-specific location recorded

Version: arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
Retrieved: 2026-10-11T21:29:34Z

catalogued

No individual claim review recorded

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.media_type

Source artifact SHA-256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Publication status
preprint
Source metadata
Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Original source ↗

No field-specific location recorded

Version: arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
Retrieved: 2026-10-11T21:29:34Z

catalogued

No individual claim review recorded

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.publication_status

Source artifact SHA-256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Retrieved at
2026-10-11T21:29:34Z
Source metadata
Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Original source ↗

No field-specific location recorded

Version: arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
Retrieved: 2026-10-11T21:29:34Z

catalogued

No individual claim review recorded

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.retrieved_at

Source artifact SHA-256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Review: date
2026-10-12
Source metadata
Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation

Original source ↗

No field-specific location recorded

Version: arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
Retrieved: 2026-10-11T21:29:34Z

catalogued

No individual claim review recorded

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Field: attributes.review.date

Source artifact SHA-256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

How to read the evidence

Sources and history

Release 2026-10-23-5347695500ed · Record review: source checked

Primary source retrieved and hashed for the 2026-10-11 cells-spatial and networks-mechanistic literature pass.

Release history

No supporting source is linked yet.

Read original source

Download this release (gzip)
Technical metadata and extraction receipts

Stable ID: lit-cells-networks-20261011-chen2026-source-chen2026

areas
single-cell; spatial-omics
contexts
research
url
https://arxiv.org/abs/2607.17227v2
artifact url
https://arxiv.org/pdf/2607.17227v2
version
arXiv 2607.17227v2, updated 2026-09-17 (v1 posted 2026-07-19)
retrieved at
2026-10-11T21:29:34Z
artifact sha256
8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da
doi
10.48550/arXiv.2607.17227
publication status
preprint
licence
CC-BY-SA-4.0
media type
application/pdf
limitations
Preprint, not peer reviewed.; Single run per cell with a fixed seed (42); no spread is printed.; Spatial dataset codes BC, MH and ME are not defined in the tables; the Figure 2 legend defines them as BC/HBCA1 (human breast cancer), MH/MHPC (mouse hippocampus) and ME/MOSTA E9.5 (mouse embryo).
evidence concerns
source id: lit-cells-networks-20261011-chen2026-source-chen2026; message: Table 4 precision and recall equal accuracy in all eight Nicheformer and CellPLM rows, as micro (global) averaging would give, but recall differs from accuracy in every scGPT-spatial, GenePT-w, scELMo and Novae row. The Methods call precision and recall 'global' without defining the averaging, so they may be averaged differently between models and should not be compared across models.; source locator: Table 4, columns 'Accuracy', 'Precision' and 'Recall'; Methods 'Cell type annotation' (page 25); artifact sha256: 8137338c6146f570a6546f30707d4244baca9dab848b32c51539665a5ba539da; review date: 2026-10-12
review
method: source-hash-verification; method note: Artifact re-downloaded from arXiv and SHA-256 matched; the archived gzip copy decompresses to the same bytes.; reviewer: claude; reviewer note: Claude review agent reviewer-cn-2, independent of the extractor lit-cells-networks; no human review claimed; date: 2026-10-12
Related records

    Suggest a correction