Geneformer and scFoundation: Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines reverses the order
- Low severity
- Not yet reviewed
- Ordering reversal
- unknown
Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines ranks Geneformer above scFoundation in 2 of 2 comparisons; 2 of 2 other studies rank scFoundation above Geneformer.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
2 papers rank scFoundation above Geneformer in 13 of their 16 comparisons and Geneformer higher in 3. Deep-learning-based gene perturbation… ranks Geneformer above scFoundation in all 2 of its comparisons.
- Rank scFoundation higher (2): HyCell CITATION.cff, Wu et al. 2025
- Rank Geneformer higher (1): Deep-learning-based gene perturbation… (disagrees)
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks scFoundation higher | Ranks Geneformer higher | Ties |
|---|---|---|---|---|
| Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines and 4 more records · DOI | Disagrees | 0 | 2 | 0 |
| HyCell CITATION.cff (commit 0dcc533c) and 17 more records · Repository | Agrees | 10 | 2 | 0 |
| Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) · DOI | Agrees | 3 | 1 | 0 |
Comparisons in Deep-learning-based gene perturbation… (2)
| Protocol | Dataset | Metric | scFoundation | Geneformer | Higher |
|---|---|---|---|---|---|
| Norman double perturbations: expression prediction for held-out double perturbations (Ahlmann-Eltze et al. 2025 Fig. 1a and Extended Data Fig. 2a) | Norman et al. 2019 Perturb-seq, K562 CRISPRa single and double perturbations (scFoundation reprocessing, Ahlmann-Eltze et al. 2025) | L2 distance (1,000 most highly expressed control genes; mean over held-out perturbations and splits), unit-unreported | 4.72 | 3.48 | Geneformer |
| Norman double perturbations: expression prediction for held-out double perturbations (Ahlmann-Eltze et al. 2025 Fig. 1a and Extended Data Fig. 2a) | Norman et al. 2019 Perturb-seq, K562 CRISPRa single and double perturbations (scFoundation reprocessing, Ahlmann-Eltze et al. 2025) | Pearson delta (1,000 most highly expressed control genes; mean over held-out perturbations and splits), unitless | 0.537 | 0.773 | Geneformer |
Comparisons in HyCell CITATION.cff (12)
| Protocol | Dataset | Metric | scFoundation | Geneformer | Higher |
|---|---|---|---|---|---|
| Annotation transfer with 15-nearest neighbours (HyCell report) | Seven held-out atlases of the HyCell report | Macro F1 (Macro-F1 of a 15-nearest-neighbour classifier trained on the other batches, over the cell types of the held-out batch, averaged over held-out batches; per atlas or averaged over the seven atlases), fraction | 0.823 | 0.782 | scFoundation |
| Annotation transfer with logistic regression (HyCell report) | Seven held-out atlases of the HyCell report | Macro F1 (Macro-F1 of a logistic-regression classifier trained on the other batches, over the cell types of the held-out batch, averaged over held-out batches; per atlas or averaged over the seven atlases), fraction | 0.849 | 0.834 | scFoundation |
| scIB batch correction (HyCell report) | Seven held-out atlases of the HyCell report | Scib batch correction score (Mean of five scIB batch-correction metrics (batch silhouette, iLISI, kBET, graph connectivity and principal-component regression), per atlas or averaged over the seven atlases), unitless | 0.436 | 0.465 | Geneformer |
| scIB biological conservation (HyCell report) | Seven held-out atlases of the HyCell report | Scib bio conservation score (Mean of seven scIB biological-conservation metrics (isolated-label score, NMI and ARI of k-means and of Leiden clusters against the cell types, cell-type silhouette and cLISI), per atlas or averaged over the seven atlases), unitless | 0.655 | 0.606 | scFoundation |
| Composite score S (HyCell report) | Asian Immune Diversity Atlas, sample (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.66 | 0.63 | scFoundation |
| Composite score S (HyCell report) | MultiVINE brain vasculature, healthy donors (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.768 | 0.756 | scFoundation |
| Composite score S (HyCell report) | Cross-tissue immune atlas, stratified sample (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.645 | 0.634 | scFoundation |
| Composite score S (HyCell report) | scIB human immune atlas, integer-count part (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.669 | 0.683 | Geneformer |
| Composite score S (HyCell report) | scIB human lung atlas, integer-count part (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.673 | 0.653 | scFoundation |
| Composite score S (HyCell report) | scIB human pancreas atlas, integer-count part (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.659 | 0.579 | scFoundation |
| Composite score S (HyCell report) | PBMC from two 10x Genomics runs (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.84 | 0.816 | scFoundation |
| Composite score S (HyCell report) | Seven held-out atlases of the HyCell report | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.702 | 0.679 | scFoundation |
Comparisons in Wu et al. 2025 (4)
What disagrees
- Other studies
- rank scFoundation above Geneformer in 13 of 16 comparisons
- This study
- ranks Geneformer above scFoundation in 2 of 2 comparisons
- Compared
- Geneformer, scFoundation
- Study that disagrees
- Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines, Source Data Extended Data Fig. 2, Ahlmann-Eltze et al. 2025, Source Data Extended Data Fig. 8, Ahlmann-Eltze et al. 2025, Source Data Fig. 1, Ahlmann-Eltze et al. 2025, Source Data Fig. 2, Ahlmann-Eltze et al. 2025
- Other studies
19 studies
Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13), HyCell CITATION.cff (commit 0dcc533c), HyCell package README (Tencent Hunyuan, commit 0dcc533c), HyCell NOTICE (commit 0dcc533c), HyCell pyproject.toml (commit 0dcc533c), HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01), HyCell weights hycell-16m weights.safetensors (commit 0dcc533c), HyCell checkpoint sidecar weights.json (commit 0dcc533c), HyInv LICENSE (MIT, commit 0dcc533c), HyInv package README (Tencent Hunyuan, commit 0dcc533c), HyInv Technical Report (Hyra Team, 2026-10-14), HyPLM-Flash engine source hyplm/engine/graphs.py (commit 0dcc533c), HyPLM-Flash engine source hyplm/engine/model.py (commit 0dcc533c), HyPLM-Flash engine source hyplm/engine/packing.py (commit 0dcc533c), HyPLM-Flash LICENSE (MIT, commit 0dcc533c), HyPLM-Flash package README (Tencent Hunyuan, commit 0dcc533c), HyPLM Technical Report (Hyra Team, 2026-10-12), Hyra Results repository LICENSE (commit 0dcc533c), Hyra Results repository README (Tencent Hunyuan, commit 0dcc533c)- Rule
- Ordering reversal; unknown
- Model identity
- Matched across studies partly through links not yet reviewed
The compared numbers
Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.
| Study | Tool | Protocol or benchmark | Metric | Value |
|---|---|---|---|---|
| Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13); Biology-driven insights into the power of single-cell foundation models | Geneformer zero-shot embeddings with OnClass (Wu et al. 2025) | Cross-atlas annotation transfer, HLCA to Tabula Sapiens, 14 shared cell types (Wu et al. 2025 Table S3) | Top-1 accuracy (Accuracy@1 over cells of the 14 shared cell types) | 0.772 |
| Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13); Biology-driven insights into the power of single-cell foundation models | Geneformer zero-shot embeddings with OnClass (Wu et al. 2025) | Cross-atlas annotation transfer, HLCA to Tabula Sapiens, 14 shared cell types (Wu et al. 2025 Table S3) | Macro F1 (test cells of the 14 shared cell types; averaged over 'all classes' (Methods 'Macro-F1'), not stated as the 14 shared types or every source-atlas type) | 0.239 |
| Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13); Biology-driven insights into the power of single-cell foundation models | scFoundation zero-shot embeddings with OnClass (Wu et al. 2025) | Cross-atlas annotation transfer, HLCA to Tabula Sapiens, 14 shared cell types (Wu et al. 2025 Table S3) | Top-1 accuracy (Accuracy@1 over cells of the 14 shared cell types) | 0.782 |
| Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13); Biology-driven insights into the power of single-cell foundation models | scFoundation zero-shot embeddings with OnClass (Wu et al. 2025) | Cross-atlas annotation transfer, HLCA to Tabula Sapiens, 14 shared cell types (Wu et al. 2025 Table S3) | Macro F1 (test cells of the 14 shared cell types; averaged over 'all classes' (Methods 'Macro-F1'), not stated as the 14 shared types or every source-atlas type) | 0.259 |
| Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13); Biology-driven insights into the power of single-cell foundation models | Geneformer zero-shot embeddings with OnClass (Wu et al. 2025) | Cross-atlas annotation transfer, Tabula Sapiens to HLCA, 14 shared cell types (Wu et al. 2025 Table S2) | Top-1 accuracy (Accuracy@1 over cells of the 14 shared cell types) | 0.665 |
| Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13); Biology-driven insights into the power of single-cell foundation models | Geneformer zero-shot embeddings with OnClass (Wu et al. 2025) | Cross-atlas annotation transfer, Tabula Sapiens to HLCA, 14 shared cell types (Wu et al. 2025 Table S2) | Macro F1 (test cells of the 14 shared cell types; averaged over 'all classes' (Methods 'Macro-F1'), not stated as the 14 shared types or every source-atlas type) | 0.087 |
| Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13); Biology-driven insights into the power of single-cell foundation models | scFoundation zero-shot embeddings with OnClass (Wu et al. 2025) | Cross-atlas annotation transfer, Tabula Sapiens to HLCA, 14 shared cell types (Wu et al. 2025 Table S2) | Top-1 accuracy (Accuracy@1 over cells of the 14 shared cell types) | 0.672 |
| Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13); Biology-driven insights into the power of single-cell foundation models | scFoundation zero-shot embeddings with OnClass (Wu et al. 2025) | Cross-atlas annotation transfer, Tabula Sapiens to HLCA, 14 shared cell types (Wu et al. 2025 Table S2) | Macro F1 (test cells of the 14 shared cell types; averaged over 'all classes' (Methods 'Macro-F1'), not stated as the 14 shared types or every source-atlas type) | 0.083 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | Annotation transfer with 15-nearest neighbours (HyCell report) | Macro F1 (Macro-F1 of a 15-nearest-neighbour classifier trained on the other batches, over the cell types of the held-out batch, averaged over held-out batches; per atlas or averaged over the seven atlases) | 0.782 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | Annotation transfer with 15-nearest neighbours (HyCell report) | Macro F1 (Macro-F1 of a 15-nearest-neighbour classifier trained on the other batches, over the cell types of the held-out batch, averaged over held-out batches; per atlas or averaged over the seven atlases) | 0.823 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | Annotation transfer with logistic regression (HyCell report) | Macro F1 (Macro-F1 of a logistic-regression classifier trained on the other batches, over the cell types of the held-out batch, averaged over held-out batches; per atlas or averaged over the seven atlases) | 0.834 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | Annotation transfer with logistic regression (HyCell report) | Macro F1 (Macro-F1 of a logistic-regression classifier trained on the other batches, over the cell types of the held-out batch, averaged over held-out batches; per atlas or averaged over the seven atlases) | 0.849 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | scIB batch correction (HyCell report) | Scib batch correction score (Mean of five scIB batch-correction metrics (batch silhouette, iLISI, kBET, graph connectivity and principal-component regression), per atlas or averaged over the seven atlases) | 0.465 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | scIB batch correction (HyCell report) | Scib batch correction score (Mean of five scIB batch-correction metrics (batch silhouette, iLISI, kBET, graph connectivity and principal-component regression), per atlas or averaged over the seven atlases) | 0.436 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | scIB biological conservation (HyCell report) | Scib bio conservation score (Mean of seven scIB biological-conservation metrics (isolated-label score, NMI and ARI of k-means and of Leiden clusters against the cell types, cell-type silhouette and cLISI), per atlas or averaged over the seven atlases) | 0.606 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | scIB biological conservation (HyCell report) | Scib bio conservation score (Mean of seven scIB biological-conservation metrics (isolated-label score, NMI and ARI of k-means and of Leiden clusters against the cell types, cell-type silhouette and cLISI), per atlas or averaged over the seven atlases) | 0.655 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.63 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.756 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.634 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.683 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.653 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.579 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.816 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | Geneformer V2 (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.679 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.66 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.768 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.645 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.669 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.673 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.659 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.84 |
| HyCell: a Compact Embedding Model for Single-Cell Transcriptomes (Hyra Team, 2026-11-01) | scFoundation (HyCell report) | Composite score S (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases) | 0.702 |
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-32ea3851946fb812 · All contested results · How findings are made and followed up