rewirebio.iobenchmarks
Evidence claim

Relevance of scTab deep-ensemble uncertainty ROC-AUC: known-type correct vs. incorrect (Group 1 vs. Group 2) to "Transfer cell-type annotations to a new dataset"

Use case
Transfer cell-type annotations to a new dataset
Assessed by
scTab deep-ensemble uncertainty ROC-AUC: known-type correct vs. incorrect (Group 1 vs. Group 2)
Relevance
Proxy evidence
Endpoint
Deep-ensemble (5-model) uncertainty-score ROC-AUC distinguishing correctly from incorrectly predicted known cell types, within one donor-held-out split of scTab's pooled 249-dataset CELLxGENE corpus.

Relevant to the use case's "identify cells that require expert review" endpoint for known-type calls (flagging the model's own low-confidence/incorrect predictions among trained types), but it is NOT an unknown-type or absent-reference detection result, and no cross-study or cross-platform transfer of this capability is established by this source.

1 reviewed evaluation

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

15 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-21-c8bea25e5abb
Property and statementOriginal source and locationReview and provenance
Citation locations
1 values
  • source id
    ucc-research-source-sctab-paper
    locator
    Methods, "Uncertainty quantification for scTab model"
Context-only references
scTab: Scaling cross-tissue single-cell annotation models

Original source ↗

ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"

Version: 10.1038/s41467-024-51059-5; published article XML retrieved 2026-09-30
Retrieved: 2026-09-30T21:23:28Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.citation_locators

Source artifact SHA-256: e61b11cd873f3d13e891e9c33bac0847070826200f8fbe0b3969d994f1c52290

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Constraints
2 values
  • Donor-level holdout within one pooled 249-dataset corpus (10x-related assays only; rare types excluded); not a whole-study or whole-platform holdout.
  • Single ROC-AUC value; no rejection-at-coverage curve or demonstrated calibration.
Context-only references
scTab: Scaling cross-tissue single-cell annotation models

Original source ↗

ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"

Version: 10.1038/s41467-024-51059-5; published article XML retrieved 2026-09-30
Retrieved: 2026-09-30T21:23:28Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.constraints

Source artifact SHA-256: e61b11cd873f3d13e891e9c33bac0847070826200f8fbe0b3969d994f1c52290

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Endpoint
Deep-ensemble (5-model) uncertainty-score ROC-AUC distinguishing correctly from incorrectly predicted known cell types, within one donor-held-out split of scTab's pooled 249-dataset CELLxGENE corpus.
Context-only references
scTab: Scaling cross-tissue single-cell annotation models

Original source ↗

ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"

Version: 10.1038/s41467-024-51059-5; published article XML retrieved 2026-09-30
Retrieved: 2026-09-30T21:23:28Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.endpoint

Source artifact SHA-256: e61b11cd873f3d13e891e9c33bac0847070826200f8fbe0b3969d994f1c52290

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Claimed field
links:assessed_by:ucc-research-protocol-sctab-uncertainty-error
Context-only references
scTab: Scaling cross-tissue single-cell annotation models

Original source ↗

ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"

Version: 10.1038/s41467-024-51059-5; published article XML retrieved 2026-09-30
Retrieved: 2026-09-30T21:23:28Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.field

Source artifact SHA-256: e61b11cd873f3d13e891e9c33bac0847070826200f8fbe0b3969d994f1c52290

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Limitations
4 values
  • This is known-type error/confidence detection, not unknown-type or absent-reference detection; it must not be combined with or treated as evidence for scTab's separate absent-type ROC-AUC (0.782, see use-case-mapping-20261006-349-2cfbe5c7674a).
  • Exact number of cells comprising the incorrect-known-type comparison group is not stated in the retrieved main text; held here as an explicit missing denominator, not invented.
  • Reference/training labels are author annotations, not independent ground truth.
  • Automated source review only; independent human scientific review remains outstanding.
Context-only references
scTab: Scaling cross-tissue single-cell annotation models

Original source ↗

ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"

Version: 10.1038/s41467-024-51059-5; published article XML retrieved 2026-09-30
Retrieved: 2026-09-30T21:23:28Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.limitations

Source artifact SHA-256: e61b11cd873f3d13e891e9c33bac0847070826200f8fbe0b3969d994f1c52290

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Pinned records
5 values
  • record id
    ucc-research-eval-sctab-uncertainty-error
    fields
    • status
    • links
    • attributes.origin
    • attributes.comparison
    sha256
    7a60dab104cd3c278409b0a5a26bd5ecdbf65bf9b6dc8a96d0f6f9975914ab0c
  • record id
    ucc-research-protocol-sctab-uncertainty-error
    fields
    • status
    • name
    • description
    • facets
    • source_ids
    • links
    • attributes
    sha256
    abcea5930a04bf7478a427e3b479d589aad2203c8685a170b4700083eca63620
  • record id
    ucc-research-result-sctab-uncertainty-error-rocauc
    fields
    • status
    • attributes.metric
    • attributes.metric_qualifier
    • attributes.printed_value
    • attributes.numeric_value
    sha256
    1e8b4e43ee42cf87a85b9cf2f34e6e7d9c6a6ef0f96bc0af7bad051200331a83
  • record id
    ucc-research-source-sctab-paper
    fields
    • status
    • attributes.artifact_sha256
    sha256
    12a4f59860912c8db02b1d56bf33636bdaccf99273cfcca9e9375ace19e54cc4
  • record id
    use-case-cell-type-annotation-transfer
    fields
    • status
    • name
    • description
    • facets
    • attributes.question
    • attributes.decision
    • attributes.inputs
    • attributes.output
    • attributes.exclusions
    • attributes.setting
    • attributes.clinical_scope
    • attributes.intended_users
    sha256
    f4ef70c0d86c1d3ea8ff8600431eb0792da0bd5f355e7137bc364a42aa284bd9
Context-only references
scTab: Scaling cross-tissue single-cell annotation models

Original source ↗

ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"

Version: 10.1038/s41467-024-51059-5; published article XML retrieved 2026-09-30
Retrieved: 2026-09-30T21:23:28Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.pins

Source artifact SHA-256: e61b11cd873f3d13e891e9c33bac0847070826200f8fbe0b3969d994f1c52290

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Rationale
Relevant to the use case's "identify cells that require expert review" endpoint for known-type calls (flagging the model's own low-confidence/incorrect predictions among trained types), but it is NOT an unknown-type or absent-reference detection result, and no cross-study or cross-platform transfer of this capability is established by this source.
Context-only references
scTab: Scaling cross-tissue single-cell annotation models

Original source ↗

ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"

Version: 10.1038/s41467-024-51059-5; published article XML retrieved 2026-09-30
Retrieved: 2026-09-30T21:23:28Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.rationale

Source artifact SHA-256: e61b11cd873f3d13e891e9c33bac0847070826200f8fbe0b3969d994f1c52290

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Reason
Add scTab's own known-type error/confidence-detection ROC-AUC (deep-ensemble uncertainty score, correct vs. incorrect known-type), independently source-checked in the additive 2026-10-06 intake (issue #349). Kept as a separate mapping from the existing Table 1a known-type macro-F1 mapping and from scTab's own unknown/absent-type detection ROC-AUC, which must not be conflated with this known-type endpoint.
Context-only references
scTab: Scaling cross-tissue single-cell annotation models

Original source ↗

ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"

Version: 10.1038/s41467-024-51059-5; published article XML retrieved 2026-09-30
Retrieved: 2026-09-30T21:23:28Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.reason

Source artifact SHA-256: e61b11cd873f3d13e891e9c33bac0847070826200f8fbe0b3969d994f1c52290

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Relevance
proxy
Context-only references
scTab: Scaling cross-tissue single-cell annotation models

Original source ↗

ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"

Version: 10.1038/s41467-024-51059-5; published article XML retrieved 2026-09-30
Retrieved: 2026-09-30T21:23:28Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.relevance

Source artifact SHA-256: e61b11cd873f3d13e891e9c33bac0847070826200f8fbe0b3969d994f1c52290

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Revision
1
Context-only references
scTab: Scaling cross-tissue single-cell annotation models

Original source ↗

ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"

Version: 10.1038/s41467-024-51059-5; published article XML retrieved 2026-09-30
Retrieved: 2026-09-30T21:23:28Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.revision

Source artifact SHA-256: e61b11cd873f3d13e891e9c33bac0847070826200f8fbe0b3969d994f1c52290

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-21-c8bea25e5abb · Record review: source checked

1 source record and release historyDownload this release (gzip)
Technical metadata and extraction receipts

Stable ID: use-case-mapping-20261006-349-6a9d63555f43

field
links:assessed_by:ucc-research-protocol-sctab-uncertainty-error
value
ucc-research-protocol-sctab-uncertainty-error
relevance
proxy
endpoint
Deep-ensemble (5-model) uncertainty-score ROC-AUC distinguishing correctly from incorrectly predicted known cell types, within one donor-held-out split of scTab's pooled 249-dataset CELLxGENE corpus.
rationale
Relevant to the use case's "identify cells that require expert review" endpoint for known-type calls (flagging the model's own low-confidence/incorrect predictions among trained types), but it is NOT an unknown-type or absent-reference detection result, and no cross-study or cross-platform transfer of this capability is established by this source.
constraints
Donor-level holdout within one pooled 249-dataset corpus (10x-related assays only; rare types excluded); not a whole-study or whole-platform holdout.; Single ROC-AUC value; no rejection-at-coverage curve or demonstrated calibration.
limitations
This is known-type error/confidence detection, not unknown-type or absent-reference detection; it must not be combined with or treated as evidence for scTab's separate absent-type ROC-AUC (0.782, see use-case-mapping-20261006-349-2cfbe5c7674a).; Exact number of cells comprising the incorrect-known-type comparison group is not stated in the retrieved main text; held here as an explicit missing denominator, not invented.; Reference/training labels are author annotations, not independent ground truth.; Automated source review only; independent human scientific review remains outstanding.
revision
1
reason
Add scTab's own known-type error/confidence-detection ROC-AUC (deep-ensemble uncertainty score, correct vs. incorrect known-type), independently source-checked in the additive 2026-10-06 intake (issue #349). Kept as a separate mapping from the existing Table 1a known-type macro-F1 mapping and from scTab's own unknown/absent-type detection ROC-AUC, which must not be conflated with this known-type endpoint.
citation locators
source id: ucc-research-source-sctab-paper; locator: Methods, "Uncertainty quantification for scTab model"
source locator
ucc-research-source-sctab-paper: Methods, "Uncertainty quantification for scTab model"
review
method: automated-source-review; reviewer: claude-sonnet; reviewer note: Claude Sonnet cell-type-annotation-transfer evidence-research worker; reviewed at: 2026-10-06T22:39:40Z; note: Source-backed primary-text transcription of the 2026-10-06 cell-type-annotation-transfer research dossier (docs/omics/evidence-research/cell-type-annotation-transfer-2026-10-06.md), corrected after an independent source check; bound by data/omics/use-case-coverage-20261006/review.json. Revised same day after a second independent review corrected the uncertainty-averaging order, removed an unsupported 'temperature-scaled' claim, replaced n_runs with ensemble_size, and removed an unsupported 'data augmentation enabled' assertion; evidence_sha256 refreshed accordingly. No ROC-AUC value or endpoint scoping changed. No new model execution, independent experimental replication, qualified human scientific review or clinical validation.
reviewed evaluations
ucc-research-eval-sctab-uncertainty-error
pins
record id: ucc-research-eval-sctab-uncertainty-error; fields: status; links; attributes.origin; attributes.comparison; sha256: 7a60dab104cd3c278409b0a5a26bd5ecdbf65bf9b6dc8a96d0f6f9975914ab0c; record id: ucc-research-protocol-sctab-uncertainty-error; fields: status; name; description; facets; source_ids; links; attributes; sha256: abcea5930a04bf7478a427e3b479d589aad2203c8685a170b4700083eca63620; record id: ucc-research-result-sctab-uncertainty-error-rocauc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 1e8b4e43ee42cf87a85b9cf2f34e6e7d9c6a6ef0f96bc0af7bad051200331a83; record id: ucc-research-source-sctab-paper; fields: status; attributes.artifact_sha256; sha256: 12a4f59860912c8db02b1d56bf33636bdaccf99273cfcca9e9375ace19e54cc4; record id: use-case-cell-type-annotation-transfer; fields: status; name; description; facets; attributes.question; attributes.decision; attributes.inputs; attributes.output; attributes.exclusions; attributes.setting; attributes.clinical_scope; attributes.intended_users; sha256: f4ef70c0d86c1d3ea8ff8600431eb0792da0bd5f355e7137bc364a42aa284bd9
Related records

Suggest a correction