Geneformer and UCE (Universal Cell Embeddings): Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines and Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) disagree
- Low severity
- Not yet reviewed
- Two-study conflict
- unknown
Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines ranks Geneformer above UCE (Universal Cell Embeddings) in 2 of 2 comparisons; Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) ranks UCE (Universal Cell Embeddings) above Geneformer in 3 of 3 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
3 papers rank Geneformer above UCE (Universal Cell Embeddings) in 6 of their 18 comparisons and UCE (Universal Cell Embeddings) higher in 9; 3 are ties. Of those papers, Wu et al. 2025 ranks UCE (Universal Cell Embeddings) higher on its own (3 of 4).
- Rank Geneformer higher (1): Deep-learning-based gene perturbation…
- Rank UCE (Universal Cell Embeddings) higher (1): Wu et al. 2025
- Mixed (1): HyCell CITATION.cff
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks Geneformer higher | Ranks UCE (Universal Cell Embeddings) higher | Ties |
|---|---|---|---|---|
| HyCell CITATION.cff (commit 0dcc533c) and 17 more records · Repository | Agrees | 4 | 6 | 2 |
| Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) · DOI | Agrees | 0 | 3 | 1 |
| Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines and 4 more records · DOI | Agrees | 2 | 0 | 0 |
Comparisons in HyCell CITATION.cff (12)
| Protocol | Dataset | Metric | Geneformer | UCE (Universal Cell Embeddings) | Higher |
|---|---|---|---|---|---|
| Annotation transfer with 15-nearest neighbours (HyCell report) | Seven held-out atlases of the HyCell report | Macro F1 (Macro-F1 of a 15-nearest-neighbour classifier trained on the other batches, over the cell types of the held-out batch, averaged over held-out batches; per atlas or averaged over the seven atlases), fraction | 0.782 | 0.814 | UCE (Universal Cell Embeddings) |
| Annotation transfer with logistic regression (HyCell report) | Seven held-out atlases of the HyCell report | Macro F1 (Macro-F1 of a logistic-regression classifier trained on the other batches, over the cell types of the held-out batch, averaged over held-out batches; per atlas or averaged over the seven atlases), fraction | 0.834 | 0.823 | Geneformer |
| scIB batch correction (HyCell report) | Seven held-out atlases of the HyCell report | Scib batch correction score (Mean of five scIB batch-correction metrics (batch silhouette, iLISI, kBET, graph connectivity and principal-component regression), per atlas or averaged over the seven atlases), unitless | 0.465 | 0.432 | Geneformer |
| scIB biological conservation (HyCell report) | Seven held-out atlases of the HyCell report | Scib bio conservation score (Mean of seven scIB biological-conservation metrics (isolated-label score, NMI and ARI of k-means and of Leiden clusters against the cell types, cell-type silhouette and cLISI), per atlas or averaged over the seven atlases), unitless | 0.606 | 0.662 | UCE (Universal Cell Embeddings) |
| Composite score S (HyCell report) | Asian Immune Diversity Atlas, sample (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.63 | 0.655 | UCE (Universal Cell Embeddings) |
| Composite score S (HyCell report) | MultiVINE brain vasculature, healthy donors (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.756 | 0.758 | Tie |
| Composite score S (HyCell report) | Cross-tissue immune atlas, stratified sample (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.634 | 0.63 | Tie |
| Composite score S (HyCell report) | scIB human immune atlas, integer-count part (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.683 | 0.715 | UCE (Universal Cell Embeddings) |
| Composite score S (HyCell report) | scIB human lung atlas, integer-count part (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.653 | 0.639 | Geneformer |
| Composite score S (HyCell report) | scIB human pancreas atlas, integer-count part (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.579 | 0.668 | UCE (Universal Cell Embeddings) |
| Composite score S (HyCell report) | PBMC from two 10x Genomics runs (HyCell report) | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.816 | 0.793 | Geneformer |
| Composite score S (HyCell report) | Seven held-out atlases of the HyCell report | Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless | 0.679 | 0.694 | UCE (Universal Cell Embeddings) |
Comparisons in Wu et al. 2025 (4)
Comparisons in Deep-learning-based gene perturbation… (2)
| Protocol | Dataset | Metric | Geneformer | UCE (Universal Cell Embeddings) | Higher |
|---|---|---|---|---|---|
| Norman double perturbations: expression prediction for held-out double perturbations (Ahlmann-Eltze et al. 2025 Fig. 1a and Extended Data Fig. 2a) | Norman et al. 2019 Perturb-seq, K562 CRISPRa single and double perturbations (scFoundation reprocessing, Ahlmann-Eltze et al. 2025) | L2 distance (1,000 most highly expressed control genes; mean over held-out perturbations and splits), unit-unreported | 3.48 | 4.28 | Geneformer |
| Norman double perturbations: expression prediction for held-out double perturbations (Ahlmann-Eltze et al. 2025 Fig. 1a and Extended Data Fig. 2a) | Norman et al. 2019 Perturb-seq, K562 CRISPRa single and double perturbations (scFoundation reprocessing, Ahlmann-Eltze et al. 2025) | Pearson delta (1,000 most highly expressed control genes; mean over held-out perturbations and splits), unitless | 0.773 | 0.61 | Geneformer |
What disagrees
- studies
- 2
- Compared
- Geneformer, UCE (Universal Cell Embeddings)
- Study that disagrees
- Other studies
- Rule
- Two-study conflict; unknown
- Model identity
- Matched across studies partly through links not yet reviewed
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-fd487a3a29f37e0e · All contested results · How findings are made and followed up