Task

Cell-type structure in frozen embeddings

Frozen cell embeddings are examined for cell structure and patient-associated batch patterns.

Sources

GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61

2 evaluations · 2 results

Overview

Datasets

The Aorta single-cell dataset is one analyzed collection, with patient and phenotype annotations.

Metrics

Adjusted Rand index compares embedding-derived clusters with patient and phenotype labels.

Allowed inputs

Gene-expression-derived cell embeddings and GenePT representations.

Sources

GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61

Evaluation procedure diagram

How it worksComputational evaluation flow

Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.

Sources

GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61

limited source coverage · Automated source review, 2026-09-16. All specifications and missing details

Results

Results are available, but no reviewed comparison panel is linked in this release.

All evaluations

2 evaluations · 2 results. Different protocols are not a single leaderboard.

Filter evaluations

Tested model or methodProtocolDatasetEvaluation setupMetricEvidence origin

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configuration	Protocol and dataset	Finding	Evidence and details
Configuration: GenePT-w	Task: Cell-type structure in frozen embeddings Dataset: Aorta single-cell dataset	0.54 Adjusted Rand Index unitless · unknown Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible	Author-reported evaluation · Source checked Methods, coverage and source GenePT-w: Cell-type structure in frozen embeddings k-means on pretrained cell embeddings; agreement with original cell-type labels. Aggregation: Not reported GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Table 2, Aorta / Cell type row, GenePT-w ARI column
Configuration: scGPT	Task: Cell-type structure in frozen embeddings Dataset: Aorta single-cell dataset	0.47 Adjusted Rand Index unitless · unknown Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible	Independent external evaluation · Source checked Methods, coverage and source scGPT: Cell-type structure in frozen embeddings k-means on pretrained cell embeddings; agreement with original cell-type labels. Aggregation: Not reported GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Table 2, Aorta / Cell type row, scGPT ARI column

Source checking is not independent reproduction. Release 2026-09-29-06401fd5b220.

Methods and evaluation design

Procedure, tasks and evaluated configurations

How it works

Evaluation methodology

The Aorta single-cell dataset is one analyzed collection, with patient and phenotype annotations. The frozen-embedding clustering analysis does not define a supervised train/test split. Random holdout used for phenotype-classifier experiments belongs to a different evaluation. Adjusted Rand index compares embedding-derived clusters with patient and phenotype labels. Geneformer, GenePT-s, scGPT and original expression measurements. The clustering comparison measures cell and patient associations in embeddings; it does not establish an independent unseen-patient supervised test.

Sources

GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61

Recorded evaluations

Each evaluation records what was tested and under which conditions.

Run instructions

No runnable recipe has been reviewed for this task. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.

A task describes a biological question. Choose a linked protocol to obtain concrete split and scoring instructions.

Strengths, limitations and unresolved questions

Strengths and limitations

Strengths and considerations

No source-reviewed explanatory claims are recorded here yet.

Limitations and conditions

Clustering agreement is not donor-held-out classification. The number of clusters is chosen to match annotated classes, and text-derived gene representations depend on the available descriptions.
Sources (4)
GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT; PMC9556750.xml; PMC7331607.xml; PMC6731122.xml · GenePT: Methods, Cell-level tasks and references 39–43; Alsaigh et al. Abstract; Chou et al. Results; Schirmer et al. Methods/Human tissue samples

Profile review details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Stable record: reported-task-4df1fb456d3deb

Specifications

Inputs, training, access and other details

Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
Property	Description and evidence
Datasets	The Aorta single-cell dataset is one analyzed collection, with patient and phenotype annotations. Sources GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61
Splits	The frozen-embedding clustering analysis does not define a supervised train/test split. Random holdout used for phenotype-classifier experiments belongs to a different evaluation. · Not reported in inspected sources Sources GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61
Metrics	Adjusted Rand index compares embedding-derived clusters with patient and phenotype labels. Sources GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61
Baselines	Geneformer, GenePT-s, scGPT and original expression measurements. Sources GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61
Leakage controls	The clustering comparison measures cell and patient associations in embeddings; it does not establish an independent unseen-patient supervised test. · Not reported in inspected sources Sources GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61
Uncertainty	The cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim. · Not reported in inspected sources Sources GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61
Entity type	Paper-specific computational evaluation protocol. Sources GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61
Organisms	Human tissue cohorts are explicitly established for the aorta, arterial plaque, osteoarthritis cartilage and multiple-sclerosis brain datasets by their original studies. GenePT also evaluates pancreas and tumor-infiltrating myeloid collections. A complete organism-by-subset census is not supplied in the GenePT cell-task description. Sources (4) GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT; PMC9556750.xml; PMC7331607.xml; PMC6731122.xml · GenePT: Methods, Cell-level tasks and references 39–43; Alsaigh et al. Abstract; Chou et al. Results; Schirmer et al. Methods/Human tissue samples
Assays	Single-cell expression with cell-type, patient and phenotype annotations. Sources GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61
Allowed inputs	Gene-expression-derived cell embeddings and GenePT representations. Sources GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61
Adaptation	Frozen-embedding geometry assessment; no task-classifier result is implied by cell separation alone. Sources GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Results: cell clustering and batch effects; Discussion; cached text lines 57, 61

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Papers and result coverage

Last literature check: 2026-09-17. Dated primary-source discovery and protocol/table screening. Source checking does not mean experimental reproduction. Only separately extracted and independently reviewed numeric batches are publishable.

Paper or primary resource	Version	Reference
GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT	PMC archival version PMC10614824.2	Read source

Historical gaps recorded on 2026-09-17

The catalogue now holds 2 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.

complete numerical transcription and independent cell review: Full primary artifact and table inventory preserved; no new numeric row is published from this audit alone.
exact checkpoint hashes and per-method scored denominators: Table labels alone do not establish these fields; do not infer checkpoint or scored count from model name or dataset size.

Search and extraction details

primary comparison tables located

Searches

GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT 10.1101/2023.10.16.562533

Evidence locations

Table 2:; XML table T6

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

23 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-29-06401fd5b220
Property and statement	Original source and location	Review and provenance
Diagram caption Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol. Individual claims	GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT Original source ↗ Results: cell clustering and batch effects; Discussion; cached text lines 57, 61 Version: PMC archival version PMC10614824.2 Retrieved: 2026-09-16T10:44:03.399274+00:00	source checked automated source review · 2026-09-16 Audit details Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: `attributes.profile.diagram.caption` Source artifact SHA-256: `230a2ec55458d9243eaeeebf3244df7409eb02d47f4b809ee56a06dcb6fdd047` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Diagram steps Input: Gene-expression-derived cell embeddings and GenePT representations. Evaluation: The frozen-embedding clustering analysis does not define a supervised train/test split. Random holdout used for phenotype-classifier experiments belongs to a different evaluation. Readout: Adjusted Rand index compares embedding-derived clusters with patient and phenotype labels. Individual claims	GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT Original source ↗ Results: cell clustering and batch effects; Discussion; cached text lines 57, 61 Version: PMC archival version PMC10614824.2 Retrieved: 2026-09-16T10:44:03.399274+00:00	source checked automated source review · 2026-09-16 Audit details Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: `attributes.profile.diagram.steps` Source artifact SHA-256: `230a2ec55458d9243eaeeebf3244df7409eb02d47f4b809ee56a06dcb6fdd047` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Diagram title Computational evaluation flow Individual claims	GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT Original source ↗ Results: cell clustering and batch effects; Discussion; cached text lines 57, 61 Version: PMC archival version PMC10614824.2 Retrieved: 2026-09-16T10:44:03.399274+00:00	source checked automated source review · 2026-09-16 Audit details Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: `attributes.profile.diagram.title` Source artifact SHA-256: `230a2ec55458d9243eaeeebf3244df7409eb02d47f4b809ee56a06dcb6fdd047` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Datasets The Aorta single-cell dataset is one analyzed collection, with patient and phenotype annotations. Individual claims	GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT Original source ↗ Results: cell clustering and batch effects; Discussion; cached text lines 57, 61 Version: PMC archival version PMC10614824.2 Retrieved: 2026-09-16T10:44:03.399274+00:00	source checked automated source review · 2026-09-16 Audit details Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: `attributes.profile.facts.0.value` Source artifact SHA-256: `230a2ec55458d9243eaeeebf3244df7409eb02d47f4b809ee56a06dcb6fdd047` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Splits The frozen-embedding clustering analysis does not define a supervised train/test split. Random holdout used for phenotype-classifier experiments belongs to a different evaluation. Individual claims	GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT Original source ↗ Results: cell clustering and batch effects; Discussion; cached text lines 57, 61 Version: PMC archival version PMC10614824.2 Retrieved: 2026-09-16T10:44:03.399274+00:00	unreported automated source review · 2026-09-16 Audit details Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: `attributes.profile.facts.1.value` Source artifact SHA-256: `230a2ec55458d9243eaeeebf3244df7409eb02d47f4b809ee56a06dcb6fdd047` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Adaptation Frozen-embedding geometry assessment; no task-classifier result is implied by cell separation alone. Individual claims	GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT Original source ↗ Results: cell clustering and batch effects; Discussion; cached text lines 57, 61 Version: PMC archival version PMC10614824.2 Retrieved: 2026-09-16T10:44:03.399274+00:00	source checked automated source review · 2026-09-16 Audit details Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: `attributes.profile.facts.10.value` Source artifact SHA-256: `230a2ec55458d9243eaeeebf3244df7409eb02d47f4b809ee56a06dcb6fdd047` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Metrics Adjusted Rand index compares embedding-derived clusters with patient and phenotype labels. Individual claims	GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT Original source ↗ Results: cell clustering and batch effects; Discussion; cached text lines 57, 61 Version: PMC archival version PMC10614824.2 Retrieved: 2026-09-16T10:44:03.399274+00:00	source checked automated source review · 2026-09-16 Audit details Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: `attributes.profile.facts.2.value` Source artifact SHA-256: `230a2ec55458d9243eaeeebf3244df7409eb02d47f4b809ee56a06dcb6fdd047` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Baselines Geneformer, GenePT-s, scGPT and original expression measurements. Individual claims	GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT Original source ↗ Results: cell clustering and batch effects; Discussion; cached text lines 57, 61 Version: PMC archival version PMC10614824.2 Retrieved: 2026-09-16T10:44:03.399274+00:00	source checked automated source review · 2026-09-16 Audit details Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: `attributes.profile.facts.3.value` Source artifact SHA-256: `230a2ec55458d9243eaeeebf3244df7409eb02d47f4b809ee56a06dcb6fdd047` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Leakage controls The clustering comparison measures cell and patient associations in embeddings; it does not establish an independent unseen-patient supervised test. Individual claims	GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT Original source ↗ Results: cell clustering and batch effects; Discussion; cached text lines 57, 61 Version: PMC archival version PMC10614824.2 Retrieved: 2026-09-16T10:44:03.399274+00:00	unreported automated source review · 2026-09-16 Audit details Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: `attributes.profile.facts.4.value` Source artifact SHA-256: `230a2ec55458d9243eaeeebf3244df7409eb02d47f4b809ee56a06dcb6fdd047` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Uncertainty The cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim. Individual claims	GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT Original source ↗ Results: cell clustering and batch effects; Discussion; cached text lines 57, 61 Version: PMC archival version PMC10614824.2 Retrieved: 2026-09-16T10:44:03.399274+00:00	unreported automated source review · 2026-09-16 Audit details Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: `attributes.profile.facts.5.value` Source artifact SHA-256: `230a2ec55458d9243eaeeebf3244df7409eb02d47f4b809ee56a06dcb6fdd047` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

View linked audit checks and correction history

Release 2026-09-29-06401fd5b220 · Record review: needs review

5 source records and release history

GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Original source · PMC archival version PMC10614824.2
GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT · Original source · PMC archival version PMC10614824.2
PMC9556750.xml · Original source · Retrieved 2026-09-16; sha256:71fd84bba280e9ac7b1009600432245122e73f6a01916a743ef4faeb56bb26f0
PMC7331607.xml · Original source · Retrieved 2026-09-16; sha256:d87b7b7a67445c04db4f1fbb0cc52914b1bccd04a23d4ff6d66e2be5373291c6
PMC6731122.xml · Original source · Retrieved 2026-09-16; sha256:2136d6c7589d2573e33c69f2b1eb0f2b76c696da06ee1c14427ccbaa45a33d5f

Download this release

Technical metadata and extraction receipts

Stable ID: reported-task-4df1fb456d3deb

areas: cells-tissues
tasks: Cell-type structure in frozen embeddings
entity level: task
version: Not reported
task: Cell-type structure in frozen embeddings
scope note: Paper-specific evaluation task; protocol completeness requires further extraction.
benchmark research: review date: 2026-09-17; status: primary_comparison_tables_located; primary sources: evidence-expansion-genept-2024-230a2ec5; inspected locators: Table 2:; XML table T6; searched queries: GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT 10.1101/2023.10.16.562533; gaps: complete numerical transcription and independent cell review: Full primary artifact and table inventory preserved; no new numeric row is published from this audit alone.; exact checkpoint hashes and per-method scored denominators: Table labels alone do not establish these fields; do not infer checkpoint or scored count from model name or dataset size.; claim scope: Dated primary-source discovery and protocol/table screening. Source checking does not mean experimental reproduction. Only separately extracted and independently reviewed numeric batches are publishable.
historical missing metadata: protocol version: not_reported_in_legacy_extract; split: not_reported_in_legacy_extract
metadata review scope: historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
legacy kinds: benchmark
entity classification: review date: 2026-09-17; rationale: This source-scoped record identifies the biological prediction task and holds its paper context. Preserve the existing task identity; exact split, model adaptation and scoring remain in linked evaluations or separate protocol records.; source ids: genept-2024; source locator: Results: cell clustering and batch effects; Discussion; cached text lines 57, 61; ambiguities: A paper- or suite-specific task may constrain some inputs or metrics; that alone does not make it interchangeable with a complete versioned protocol. No protocol equivalence is inferred.; Some legacy profile Entity type facts use the generic phrase computational evaluation protocol. That boilerplate is not sufficient to establish a single fixed protocol identity or to merge this task with another protocol record.

Related records

dataset: Aorta single-cell dataset
benchmark: GenePT-w: Cell-type structure in frozen embeddings
benchmark: scGPT: Cell-type structure in frozen embeddings

Suggest a correction