Protocol

Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset

INTRA-dataset evaluation (same single dataset, stratified 5-fold cross-validation); NOT cross-study or cross-platform transfer. The source separately reports genuine inter-dataset (cross-platform/cross-study) experiments (Fig. 4 brain, Fig. 5 pancreatic) that are explicitly out of scope for this protocol and are not ingested here. Median F1-score and, for rejection-capable classifiers, the percentage of cells left "unlabeled" are reported for the Baron Human dataset specifically.

4 evaluations · 7 results

Overview

Consult the linked sources for architecture or protocol details. Missing evidence is not evidence of a missing capability.

4 recorded evaluations, 7 metric rows. A comparison chart has not yet been validated for these results. The table retains the individual findings and their sources.

View coverage and remaining gaps across all benchmarks

Results

Results are available, but no reviewed comparison panel is linked in this release.

All evaluations

4 evaluations · 7 results. Different protocols are not a single leaderboard.

Filter evaluations

Tested model or methodProtocolDatasetEvaluation setupMetricEvidence origin

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configuration	Protocol and dataset	Finding	Evidence and details
Configuration: scmapcell — Abdelaal et al. 2019 Table 1	Protocol: Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset Dataset: Baron Human pancreatic scRNA-seq dataset (Abdelaal et al. 2019, Table 2)	0.984 median-f1 fraction · higher Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible	Author-reported evaluation · Source checked Methods, coverage and source abdelaal baron human scmapcell Not reported Aggregation: Not reported A comparison of automatic cell identification methods for single-cell RNA sequencing data · Quoted verbatim, Results, "All classifiers perform well in intra-dataset experiments": "for the Baron Human dataset, the median F1-score for SVM rejection, scmapcell, scPred, and SVM is 0.991, 0.984, 0.981, and 0.980, respectively (Fig. 1a)." Fig. 1a, scmapcell row.
Configuration: scmapcell — Abdelaal et al. 2019 Table 1	Protocol: Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset Dataset: Baron Human pancreatic scRNA-seq dataset (Abdelaal et al. 2019, Table 2)	4.2% pct-unlabeled percent · unknown Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible	Author-reported evaluation · Source checked Methods, coverage and source abdelaal baron human scmapcell Not reported Aggregation: Not reported A comparison of automatic cell identification methods for single-cell RNA sequencing data · Quoted verbatim, same location: "SVM rejection, scmapcell, and scPred assigned 1.5%, 4.2%, and 10.8% of the cells, respectively, as unlabeled while SVM (without rejection) classified 100% of the cells with a median F1-score of 0.98 (Fig. 1b)." Fig. 1b, scmapcell row.
Configuration: scPred — Abdelaal et al. 2019 Table 1	Protocol: Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset Dataset: Baron Human pancreatic scRNA-seq dataset (Abdelaal et al. 2019, Table 2)	0.981 median-f1 fraction · higher Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible	Author-reported evaluation · Source checked Methods, coverage and source abdelaal baron human scpred Not reported Aggregation: Not reported A comparison of automatic cell identification methods for single-cell RNA sequencing data · Quoted verbatim, Results, "All classifiers perform well in intra-dataset experiments": "for the Baron Human dataset, the median F1-score for SVM rejection, scmapcell, scPred, and SVM is 0.991, 0.984, 0.981, and 0.980, respectively (Fig. 1a)." Fig. 1a, scPred row.
Configuration: scPred — Abdelaal et al. 2019 Table 1	Protocol: Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset Dataset: Baron Human pancreatic scRNA-seq dataset (Abdelaal et al. 2019, Table 2)	10.8% pct-unlabeled percent · unknown Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible	Author-reported evaluation · Source checked Methods, coverage and source abdelaal baron human scpred Not reported Aggregation: Not reported A comparison of automatic cell identification methods for single-cell RNA sequencing data · Quoted verbatim, same location: "SVM rejection, scmapcell, and scPred assigned 1.5%, 4.2%, and 10.8% of the cells, respectively, as unlabeled while SVM (without rejection) classified 100% of the cells with a median F1-score of 0.98 (Fig. 1b)." Fig. 1b, scPred row.
Configuration: SVM (no rejection) — Abdelaal et al. 2019 Table 1	Protocol: Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset Dataset: Baron Human pancreatic scRNA-seq dataset (Abdelaal et al. 2019, Table 2)	0.98 median-f1 fraction · higher Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible	Author-reported evaluation · Source checked Methods, coverage and source abdelaal baron human svm Not reported Aggregation: Not reported A comparison of automatic cell identification methods for single-cell RNA sequencing data · Quoted verbatim, Results, "All classifiers perform well in intra-dataset experiments": "for the Baron Human dataset, the median F1-score for SVM rejection, scmapcell, scPred, and SVM is 0.991, 0.984, 0.981, and 0.980, respectively (Fig. 1a)." Fig. 1a, SVM row. (Also restated as 0.98 alongside the unlabeled-percentage sentence, Fig. 1b.)
Configuration: SVMrejection — Abdelaal et al. 2019 Table 1	Protocol: Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset Dataset: Baron Human pancreatic scRNA-seq dataset (Abdelaal et al. 2019, Table 2)	0.991 median-f1 fraction · higher Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible	Author-reported evaluation · Source checked Methods, coverage and source abdelaal baron human svm rejection Not reported Aggregation: Not reported A comparison of automatic cell identification methods for single-cell RNA sequencing data · Quoted verbatim, Results, "All classifiers perform well in intra-dataset experiments": "for the Baron Human dataset, the median F1-score for SVM rejection, scmapcell, scPred, and SVM is 0.991, 0.984, 0.981, and 0.980, respectively (Fig. 1a)." Fig. 1a, SVM rejection row.
Configuration: SVMrejection — Abdelaal et al. 2019 Table 1	Protocol: Abdelaal et al. 2019 intra-dataset 5-fold CV: Baron Human pancreatic dataset Dataset: Baron Human pancreatic scRNA-seq dataset (Abdelaal et al. 2019, Table 2)	1.5% pct-unlabeled percent · unknown Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible	Author-reported evaluation · Source checked Methods, coverage and source abdelaal baron human svm rejection Not reported Aggregation: Not reported A comparison of automatic cell identification methods for single-cell RNA sequencing data · Quoted verbatim, same location: "SVM rejection, scmapcell, and scPred assigned 1.5%, 4.2%, and 10.8% of the cells, respectively, as unlabeled while SVM (without rejection) classified 100% of the cells with a median F1-score of 0.98 (Fig. 1b)." Fig. 1b, SVM rejection row.

Source checking is not independent reproduction. Release 2026-10-06-161b59a1d02c.

Methods and evaluation design

Procedure, tasks and evaluated configurations

Recorded evaluations

Each evaluation records what was tested and under which conditions.

Baseline coverage

Reference methods help show what a model adds beyond simple controls. We track a null control and a conventional method for each protocol.

0 of 2 active baseline roles have published Rewire measurements in this release. Measurements on a selected protocol do not establish coverage of an entire suite.

No execution recipe linked to this protocol. Recipe availability does not establish a completed evaluation.

Author-reported evaluations: 4

Literature evidence is not a Rewire measurement. Executed but unpublished runs and private review status are not included.

Null control

Proposed control: requires review

Training-set class prior where supervised fitting is permitted

Protocol-specific applicability, permitted inputs, access, split, evaluator and execution requirements need review before implementation or execution.

This is a suggested selection rule, not a validated method or a measured score.

Conventional reference

Proposed control: requires review

Regularised classifier on simple permitted features, or protocol's conventional reference

Protocol-specific applicability, permitted inputs, access, split, evaluator and execution requirements need review before implementation or execution.

This is a suggested selection rule, not a validated method or a measured score.

Protocol coverage CSV · Model evaluation matrix · Source table · Release and checksums

Coverage is derived from release 2026-10-06-161b59a1d02c. Source citations describe the original records; they do not validate an unreviewed baseline proposal. No results have been generated by this audit.

Run instructions

No runnable recipe has been reviewed for this protocol. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.

Strengths, limitations and unresolved questions

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

0 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-06-161b59a1d02c
Property and statement	Original source and location	Review and provenance

No evidence rows match these filters. Choose another scope or clear the search.

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

View linked audit checks and correction history

Release 2026-10-06-161b59a1d02c · Record review: source checked

1 source records and release history

A comparison of automatic cell identification methods for single-cell RNA sequencing data · Original source · 10.1186/s13059-019-1795-z; published article XML

Download this release

Technical metadata and extraction receipts

Stable ID: ucc-research-protocol-abdelaal-baron-human-intra

review: method: automated_source_review; actor: Claude Sonnet cell-type-annotation-transfer evidence-research worker; reviewed at: 2026-10-06T23:28:54Z; note: Source-backed primary-text transcription of Abdelaal et al. 2019 (Genome Biology). Re-fetched directly from Europe PMC at 2026-10-06T23:28:54Z (this review pass; not the earlier bounded-window estimate), byte-identical to the already-cached copy (SHA-256 unchanged), and archived as a committed artifact at data/omics/use-case-coverage-20261006/research/artifacts/abdelaal-2019-pmc6734286-fulltext.xml.gz. No new model execution, independent experimental replication, or qualified human scientific review.
source locator: Methods, "Intra-dataset classification"; Results, "Benchmarking automatic cell identification methods (intra-dataset evaluation)" > "All classifiers perform well in intra-dataset experiments"; Fig. 1a,b; Table 1 (method identities); Table 2 (dataset population)
limitations: INTRA-dataset only (same single dataset, stratified 5-fold CV); does not establish cross-study or cross-platform transfer. Confirmed from the source's own Methods section, which explicitly distinguishes this "intra-dataset" design from a separate "inter-dataset" design describing training on one dataset/combination and testing on a different one.; Median F1-score is a summary statistic across cell populations within the dataset (as stated in the source), not a per-population breakdown; per-population values are not extracted here.; Rejection-option percentages (cells left unlabeled) are reported only for SVMrejection, scmapcell and scPred; SVM (no rejection) is quoted as classifying "100% of the cells" (0% unlabeled) and is not given a separate rejection-percentage result record in this intake.; Reference labels for this dataset are the original Baron et al. study's author-assigned cell-population annotations, not independently adjudicated ground truth; the source does not claim otherwise.; Exact per-fold cell counts and the fold manifest are not stated in the retrieved main text.; Automated source review only; independent human scientific review remains outstanding.; The dataset's 8,569-cell figure (Table 2) is the overall dataset size, not independently confirmed as the exact scored denominator for any individual classifier's median F1-score or unlabeled-percentage result; per-classifier/per-fold scored counts are not stated in the retrieved main text.; Percentage of cells left unlabeled is a coverage/rejection-rate figure, not a performance metric with a universal better/worse direction on its own: a classifier can lower its unlabeled percentage by accepting more low-confidence calls, which can raise or lower its own median F1-score depending on those calls' correctness. It must be read jointly with the same classifier's median F1-score (an explicit F1/rejection tradeoff), not interpreted in isolation. metric_direction is recorded as "unknown" for this reason, per the schema's own three-value higher/lower/unknown contract (services/omics/src/validation.ts).

Related records

dataset: Baron Human pancreatic scRNA-seq dataset (Abdelaal et al. 2019, Table 2)
protocol: abdelaal baron human scmapcell
protocol: abdelaal baron human scpred
protocol: abdelaal baron human svm
protocol: abdelaal baron human svm rejection

Suggest a correction