rewirebio.iobenchmarks

Use caseCells and tissuesResearch only

Transfer cell-type annotations to a new dataset

Which annotation workflow can label my new dataset reliably and recognise unsupported cell populations?

What the evidence shows

In Wu et al. 2025 (scFM-Bench), whose authors did not develop the scored models, OnClass classifiers on zero-shot embeddings from six single-cell foundation models transferred cell-type labels between two public atlases, scored on cells of the 14 cell types the atlases share and against the atlases' own annotations, which are not independent ground truth.

From Tabula Sapiens to the lung atlas (HLCA), Accuracy@1 ranged from 0.653 (LangCell) to 0.785 (scCello) but macro-F1 only from 0.08 to 0.105; from HLCA to Tabula Sapiens, Accuracy@1 ranged from 0.614 (scGPT) to 0.857 (scCello) and macro-F1 from 0.209 to 0.283 (UCE). The best pairwise ensembles raised macro-F1 to 0.164 (Geneformer + scCello) and 0.303 (UCE + scCello) without exceeding the best single model's accuracy. The source does not state which classes macro-F1 averages over, there is one run per configuration with no spread, no conventional baseline appears in these tables, and rejection of cell types absent from the reference is reported only in figures, so this evidence does not show whether a workflow can flag unsupported populations.

The page's other evidence (Abdelaal et al. 2019, scTab) uses separate protocols and is not pooled with these values.

You bring
  • Query count data with tissue, disease, donor and assay metadata
  • A versioned reference and a defined cell-label hierarchy
  • Independent marker, protein or expert evidence where available
  • Requirements for coverage, uncertainty and resource use
You want
Cell-type labels at the requested ontology level, confidence estimates and an explicit unassigned group.
Evidence in this release
6 comparisons shown · 4 held for review · 17 tools · 36 configurations
Kept up to date
Evidence last checked 9 October 2026 · re-checked monthly, next due 9 November 2026

Baseline

scmap-cell, linear SVM and logistic regression. Manual marker-based annotation, the pre-machine-learning practice, has no method record, so the simplest measured comparators are named instead: nearest-reference projection (scmap-cell) and linear classifiers trained on labelled expression. Abdelaal et al. 2019 found the linear SVM the best performer overall among 22 methods, and scTab uses a tuned logistic regression as its reference model.

Machine-learning methods are compared with the baseline on the same data in 2 tables: scTab Table 1a: seeded model fits, Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset. Baseline rows are marked in each table.

Tools compared

Comparisons

Each table is one study or protocol with its own truth set and scoring. Compare tools within a table, not across tables.

scTab Table 1a: seeded model fits

Proxy evidenceAuthor-reported

Donor-held-out, ontology-adjusted macro F1 for known cell types.

Sorted by Macro F1 (higher is better). The best value in each column is highlighted.

ToolMacro F1
scTab — scTab Table 1machine learning0.8295 ± 0.0007 (best in column)
XGBoost — scTab Table 1machine learning0.8127 ± 0.0005
MLP — scTab Table 1machine learning0.7971 ± 0.0012
Optimized linear logistic regression — scTab Table 1Baselinemachine learning0.7848 ± 0.0001
CIForm — scTab Table 1machine learning0.7660
UCE frozen embeddings + classifier — scTab Table 1foundation model0.7611 ± 0.0018
scGPT fine-tuned — scTab Table 1foundation model0.7490
CellTypist — scTab Table 1machine learning0.7304 ± 0.0015
scGPT frozen embeddings + logistic regression — scTab Table 1foundation model0.7301 ± 0.0035

± is a standard deviation.

Caveats and method
  • Unknown-type rejection ROC appears in Supplementary Figure 4 but numerical curve labels are not extracted here.
  • Reference labels are author annotations, not independent ground truth.
  • Checkpoint and split hashes unextracted; no deployment calibration claim.
  • Automated source review only; independent human scientific review remains outstanding.

Protocol: scTab Table 1a: seeded model fits

Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset

Proxy evidenceAuthor-reported

Intra-dataset (same single Baron Human dataset, stratified 5-fold cross-validation, identical folds across classifiers) median F1-score and, for rejection-capable classifiers, percentage of cells left unlabeled.

Sorted by Median F1 (higher is better). The best value in each column is highlighted.

Scroll sideways to see all 2 columns.

ToolMedian F1Unassigned rate
SVMrejection — Abdelaal et al. 2019 Table 1Baselinemachine learning0.991 (best in column)1.5%
scmapcell — Abdelaal et al. 2019 Table 1Baselineconventional0.9844.2%
scPred — Abdelaal et al. 2019 Table 1machine learning0.98110.8%
SVM (no rejection) — Abdelaal et al. 2019 Table 1Baselinemachine learning0.980Not reported
Caveats and method
  • Does not establish cross-study or cross-platform transfer; the source's own inter-dataset (cross-platform/cross-study) experiments (Fig. 4 brain, Fig. 5 pancreatic) are not ingested in this mapping.
  • Median F1-score is a per-dataset summary statistic across cell populations, not a per-population breakdown.
  • SVM (no rejection) is quoted as classifying "100% of the cells" (0% unlabeled by implication) but is not given its own rejection-percentage result record in this intake.
  • Reference labels are the original Baron et al. study's author-assigned cell-population annotations, not independently adjudicated ground truth.
  • Exact per-fold cell counts and the fold manifest are not stated in the retrieved main text.
  • 8,569 cells is the Table 2 Baron Human dataset-size figure; it is not independently confirmed as the exact number of cells scored for any individual classifier's median F1 or unlabeled-percentage result.
  • Percentage unlabeled is a coverage/rejection-rate tradeoff against the same classifier's own median F1-score, not a standalone performance figure with a universal better/worse direction; it must be read jointly with that classifier's F1, not in isolation. The corresponding result records carry metric_direction: "unknown" for this reason, per the schema's higher/lower/unknown contract.
  • Automated source review only; independent human scientific review remains outstanding.

Protocol: Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset

Cross-atlas label transfer with foundation-model embeddings (Wu et al. 2025)

Proxy evidenceIndependent study

Macro F1 (higher is better), with the other metrics below each value. Showing the top 10 of 23 rows. The best value in each column is highlighted.

Scroll sideways to see all 2 columns.

ToolTabula Sapiens to HLCAHLCA to Tabula Sapiens
Geneformer + scCellofoundation model0.164 (best in column)Top-1 accuracy 0.760.288Top-1 accuracy 0.828
Logit aggregationfoundation model0.163Top-1 accuracy 0.7530.285Top-1 accuracy 0.82
Geneformer + LangCellfoundation model0.162Top-1 accuracy 0.750.285Top-1 accuracy 0.814
Geneformer + scGPTfoundation model0.161Top-1 accuracy 0.7560.285Top-1 accuracy 0.816
Geneformer + UCEfoundation model0.161Top-1 accuracy 0.7470.290Top-1 accuracy 0.824
Geneformer + scFoundationfoundation model0.156Top-1 accuracy 0.7440.281Top-1 accuracy 0.816
scCello + scGPTfoundation model0.130Top-1 accuracy 0.7750.275Top-1 accuracy 0.793
scCello + scFoundationfoundation model0.128Top-1 accuracy 0.7310.286Top-1 accuracy 0.827
scCello + UCEfoundation model0.126Top-1 accuracy 0.7360.303 (best in column)Top-1 accuracy 0.852
LangCell + scCellofoundation model0.125Top-1 accuracy 0.7470.277Top-1 accuracy 0.818
Majority votingfoundation model0.119Top-1 accuracy 0.7580.280Top-1 accuracy 0.83
scGPT + UCEfoundation model0.118Top-1 accuracy 0.7250.270Top-1 accuracy 0.802
LangCell + scGPTfoundation model0.112Top-1 accuracy 0.7190.253Top-1 accuracy 0.769
scFoundation + scGPTfoundation model0.112Top-1 accuracy 0.7180.261Top-1 accuracy 0.775
scFoundation + UCEfoundation model0.112Top-1 accuracy 0.7030.275Top-1 accuracy 0.809
LangCell + scFoundationfoundation model0.111Top-1 accuracy 0.7080.266Top-1 accuracy 0.79
LangCell + UCEfoundation model0.111Top-1 accuracy 0.7130.275Top-1 accuracy 0.808
scCellofoundation model0.105Top-1 accuracy 0.7850.271Top-1 accuracy 0.857
UCEfoundation model0.089Top-1 accuracy 0.6690.283Top-1 accuracy 0.825
Geneformerfoundation model0.087Top-1 accuracy 0.6650.239Top-1 accuracy 0.772
scFoundationfoundation model0.083Top-1 accuracy 0.6720.259Top-1 accuracy 0.782
scGPTfoundation model0.081Top-1 accuracy 0.6740.209Top-1 accuracy 0.614
LangCellfoundation model0.080Top-1 accuracy 0.6530.244Top-1 accuracy 0.766
Show all 23 rowsShow the top 10 rows only
Caveats and method

What each column measures:

  • Tabula Sapiens to HLCA: Accuracy@1 and macro-F1 for OnClass on six scFM embeddings, 15 pairwise and 2 full ensembles, transferring labels from Tabula Sapiens to HLCA on the 14 shared cell types
  • HLCA to Tabula Sapiens: Accuracy@1 and macro-F1 for OnClass on six scFM embeddings, 15 pairwise and 2 full ensembles, transferring labels from HLCA to Tabula Sapiens on the 14 shared cell types
  • Truth labels are the atlases' own annotations ('cell_ontology_class' for Tabula Sapiens, 'cell_type' for HLCA core; Table 2), reduced to leaf terms with at least 10 cells; the use case excludes treating atlas labels as infallible ground truth.
  • Zero-shot embeddings with one classifier (OnClass); no fine-tuning.
  • The OnClass classifier is trained on the 20% training split of the source atlas (all 120 or 37 source types) and applied to every cell of the 14 shared types, so predictions onto source-only types count as errors; macro-F1 is averaged over 'all classes', and the class set is not stated.
  • No rejection or unassigned endpoint is in Tables S2 and S3. Novel-type detection (AUROC, AUPRC; Fig. S22) and Accuracy@FPR (Fig. 3e, Fig. S23) are figure-only and not extracted.
  • Conventional baselines are not in Tables S2 and S3; the text reports scVI for cross-dataset transfer (Fig. 3b) and logistic regression for intra-dataset validation (Fig. 3a), figures only.
  • One run per configuration; no seeds or spread printed.
  • Results paragraph 4 reverses the full-ensemble comparison relative to both tables (source limitation); table values are used.

Protocols: Cross-atlas annotation transfer, Tabula Sapiens to HLCA, 14 shared cell types (Wu et al. 2025 Table S2), Cross-atlas annotation transfer, HLCA to Tabula Sapiens, 14 shared cell types (Wu et al. 2025 Table S3)

scTab deep-ensemble uncertainty ROC-AUC: known-type vs. absent-type (Group 1 vs. Group 3)

Proxy evidenceAuthor-reported

Deep-ensemble (5-model) uncertainty-score ROC-AUC distinguishing correctly predicted known cell types from cell types entirely absent from training, within one donor-held-out split of scTab's pooled 249-dataset CELLxGENE corpus.

Single reported result.

scTab — scTab Table 1machine learning

AUROC (known vs unknown cell type)
0.782
Caveats and method
  • This is unknown/absent-type detection only; it must not be combined with or treated as evidence for scTab's separate known-type error-detection ROC-AUC (0.891, see use-case-mapping-20261006-349-6a9d63555f43).
  • Exact number of cell types and cells comprising the absent-type comparison group is not stated in the retrieved main text; held here as an explicit missing denominator, not invented.
  • Reference/training labels are author annotations, not independent ground truth.
  • Automated source review only; independent human scientific review remains outstanding.

Protocol: scTab deep-ensemble uncertainty ROC-AUC: known-type vs. absent-type (Group 1 vs. Group 3)

scTab deep-ensemble uncertainty ROC-AUC: known-type correct vs. incorrect (Group 1 vs. Group 2)

Proxy evidenceAuthor-reported

Deep-ensemble (5-model) uncertainty-score ROC-AUC distinguishing correctly from incorrectly predicted known cell types, within one donor-held-out split of scTab's pooled 249-dataset CELLxGENE corpus.

Single reported result.

scTab — scTab Table 1machine learning

AUROC (known vs unknown cell type)
0.891
Caveats and method
  • This is known-type error/confidence detection, not unknown-type or absent-reference detection; it must not be combined with or treated as evidence for scTab's separate absent-type ROC-AUC (0.782, see use-case-mapping-20261006-349-2cfbe5c7674a).
  • Exact number of cells comprising the incorrect-known-type comparison group is not stated in the retrieved main text; held here as an explicit missing denominator, not invented.
  • Reference/training labels are author annotations, not independent ground truth.
  • Automated source review only; independent human scientific review remains outstanding.

Protocol: scTab deep-ensemble uncertainty ROC-AUC: known-type correct vs. incorrect (Group 1 vs. Group 2)

Abdelaal et al. 2019 inter-dataset brain comparison: VISp/ALM/MTG, 34-population annotation (Figure S10 Panel B)

Proxy evidenceAuthor-reported

Inter-dataset (cross-dataset, cross-species where applicable) SVMrejection conventional rejection-option baseline: percentage of test-set cells left unlabeled across all 9 VISp/ALM/MTG train-test combinations, 34-cell-population annotation level.

Single reported result.

SVMrejection — Abdelaal et al. 2019 Table 1Baselinemachine learning

Unassigned rate
81.1%
Caveats and method
  • Scored denominator is UNREPORTED in the retrieved main text and supplement. Table 2's raw per-dataset cell counts (VISp 12,832; ALM 8,758; MTG 14,636) are recorded on the linked dataset records as context only, not a confirmed denominator, and must not be used to back-calculate an implied rejected-cell count.
  • metric_direction is "unknown" for every result under this mapping's protocol: percentage unlabeled is a coverage/rejection-rate figure, not a standalone performance metric with one universal better/worse direction.
  • Not a known-type accuracy claim, not an unknown/absent-type-detection accuracy claim, and not a superiority claim over any other classifier; no comparison across methods is asserted by this mapping.
  • Figure S9 Panel A (PBMC) carries no printed digit labels in the source PDF at any resolution checked, and separately shows an unresolved conflict between its own caption ("median F1-score"), the page's only legend ("Unlabeled (%)"), and the figure's overall title ("Percentage of unlabeled cells"); nothing from Figure S9 is part of this mapping.
  • Reference labels are the original brain-atlas studies' author-assigned cell-population annotations, not independently adjudicated ground truth.
  • VISp and ALM share GSE115746. MTG is a separate human single-nucleus dataset; combinations involving MTG include cross-species data. Some two-dataset training combinations still share the mouse study with the test dataset, so independent-study holdout must not be inferred for all nine combinations.
  • Automated source review only; independent human scientific review remains outstanding.
  • Predates scTab, CellTypist, Azimuth, scANVI and scParadise; does not evaluate them. Does not measure or substitute for the common-protocol cross-study benchmark gap tracked separately at rewire-benchmarks #27.

Protocol: Abdelaal et al. 2019 inter-dataset brain comparison: VISp/ALM/MTG, 34-population annotation (Figure S10 Panel B)

Held for review

Recorded but not shown as evidence until the reason below is resolved.

Harmonised single-cell and spatial foundation model benchmark (Chen et al. 2026) (ATAA held-out annotation)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-chen2026-eval-annotation-ataa-cellplm, lit-cells-networks-20261011-chen2026-eval-annotation-ataa-genept-w, lit-cells-networks-20261011-chen2026-eval-annotation-ataa-nicheformer, lit-cells-networks-20261011-chen2026-eval-annotation-ataa-scelmo, lit-cells-networks-20261011-chen2026-eval-annotation-ataa-scgpt-spatial); no evaluation on the protocol is eligible evidence.

Harmonised single-cell and spatial foundation model benchmark (Chen et al. 2026) (BMMC held-out annotation)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-chen2026-eval-annotation-bmmc-cellplm, lit-cells-networks-20261011-chen2026-eval-annotation-bmmc-genept-w, lit-cells-networks-20261011-chen2026-eval-annotation-bmmc-nicheformer, lit-cells-networks-20261011-chen2026-eval-annotation-bmmc-scelmo, lit-cells-networks-20261011-chen2026-eval-annotation-bmmc-scgpt-spatial); no evaluation on the protocol is eligible evidence.

Harmonised single-cell and spatial foundation model benchmark (Chen et al. 2026) (HS held-out annotation)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-chen2026-eval-annotation-hs-cellplm, lit-cells-networks-20261011-chen2026-eval-annotation-hs-genept-w, lit-cells-networks-20261011-chen2026-eval-annotation-hs-nicheformer, lit-cells-networks-20261011-chen2026-eval-annotation-hs-scelmo, lit-cells-networks-20261011-chen2026-eval-annotation-hs-scgpt-spatial); no evaluation on the protocol is eligible evidence.

Harmonised single-cell and spatial foundation model benchmark (Chen et al. 2026) (SEAAD held-out annotation)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (lit-cells-networks-20261011-chen2026-eval-annotation-seaad-cellplm, lit-cells-networks-20261011-chen2026-eval-annotation-seaad-genept-w, lit-cells-networks-20261011-chen2026-eval-annotation-seaad-nicheformer, lit-cells-networks-20261011-chen2026-eval-annotation-seaad-novae, lit-cells-networks-20261011-chen2026-eval-annotation-seaad-scelmo); no evaluation on the protocol is eligible evidence.

Not covered yet

  • Automated source review only; independent human scientific review remains outstanding.
  • Checkpoint and split hashes unextracted; no deployment calibration claim.
  • Reference labels are author annotations, not independent ground truth.
  • Unknown-type rejection ROC appears in Supplementary Figure 4 but numerical curve labels are not extracted here.
  • No independent comparison found in this pass prints per-method rejection of unseen cell types (unknown-population F1 or AUROC) for CellTypist, scANVI, SingleR, Azimuth or scArches; such results appear in developer method papers or in figures.
  • Wu et al. 2025's novel-cell-type results (AUROC and AUPRC across unseen ratios, and Accuracy@FPR) are figure-only, as are its scVI cross-dataset baseline and its logistic regression intra-dataset baseline.
  • Boiarsky et al. 2023 (logistic regression against scBERT and scGPT) prints its tables as images in the preprint; the published version was not retrieved.
  • The extracted cross-atlas transfer uses the atlases' own labels as truth, scores only cells of the 14 shared cell types, and does not state which classes its macro-F1 averages over.

Contribute evidence or propose a correction

Question, sources and review

Who this is for, setting and exclusions

Choose an annotation workflow and reference, accept supported labels, and identify cells that require expert review or additional measurements.

Who this is for
  • Single-cell analysts
  • Cell atlas researchers
Biological setting
Research annotation of a new single-cell or single-nucleus cohort. Transfer across donors, studies, disease contexts and technologies is assessed separately.
Research setting
Research only. Annotation accuracy does not establish the validity of a clinical diagnostic classifier.

Outside this use case

  • Atlas labels are not infallible ground truth.
  • Embedding separation and batch mixing do not establish correct cell identity.
  • Annotation, batch integration and discovery of new disease states require separate evaluations.
Evidence collection plan
Evidence collection planCollecting evidence

Already on record: 6 comparisons shown · 4 held for review · 17 tools, in the evaluated evidence above. The status above describes only the specific comparison in this plan, which remains open; it does not mean no evidence has been collected. The plan defines a comparison to investigate; it does not establish model performance or suitability.

Comparison question

Does a method improve known-type annotation and unknown-type recognition at matched coverage over conventional methods across independent studies?

Baselines, outcomes and validation requirements

Baselines to include

  • Marker-based and nearest-reference annotation
  • Appropriate pinned Azimuth, CellTypist or scANVI workflows
  • Embeddings with a simple classifier where applicable

Outcomes to measure

  • Class-specific and hierarchy-aware errors
  • Rare-type and absent-reference performance
  • Calibration and accuracy as uncertain cells are left unassigned
  • Resource use and transfer across studies and platforms

Validation requirements

  • Hold out donors, whole studies and platforms as required by the intended query.
  • Audit duplicated cells, donors and labels across references and pretraining data.
  • Use independent expert or orthogonal label checks and preserve uncertainty in reference labels.
  • Match ontology level and coverage; keep integration and disease-state discovery separate.

Next collection task

Close the remaining endpoint, independence and comparator gaps documented in the 30 September 2026 audit and linked article issue; obtain qualified scientific review.

Related articles

Read the background, methods and practical considerations behind this question.

Sources, review and downloads

Automated source review · 2026-09-30 · Codex source curation, independent worker cross-review and root integration review

Source-backed literature curation with independent automated transcription and scope checks. No new model execution, independent experimental replication, qualified human scientific review or clinical validation.

Release 2026-11-02-a7f03c9e55a5 · question use-case-cell-type-annotation-transfer

Use-case input digest 43a80aef5473f1a49c013b6d46b720b850ff5a0d18dcbb741dcdeed1ed082db1

Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)