rewirebio.iobenchmarks
Contested result

Geneformer and UCE (Universal Cell Embeddings): Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines and Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines ranks Geneformer above UCE (Universal Cell Embeddings) in 2 of 2 comparisons; Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) ranks UCE (Universal Cell Embeddings) above Geneformer in 3 of 3 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

3 papers rank Geneformer above UCE (Universal Cell Embeddings) in 6 of their 18 comparisons and UCE (Universal Cell Embeddings) higher in 9; 3 are ties. Of those papers, Wu et al. 2025 ranks UCE (Universal Cell Embeddings) higher on its own (3 of 4).

Papers grouped by which system they rank higher: Geneformer or UCE (Universal Cell Embeddings)Rank Geneformer higher: Deep-learning-based gene perturbation…. Rank UCE (Universal Cell Embeddings) higher: Wu et al. 2025. Mixed: HyCell CITATION.cff.Rank Geneformer higherMixedRank UCE (Universal Cell Embe…Deep-learning-basedgene perturbation…Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselinesHyCellCITATION.cffHyCell CITATION.cff (commit 0dcc533c)Wu et al. 2025Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13)
  • Rank Geneformer higher (1): Deep-learning-based gene perturbation…
  • Rank UCE (Universal Cell Embeddings) higher (1): Wu et al. 2025
  • Mixed (1): HyCell CITATION.cff

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Geneformer or UCE (Universal Cell Embeddings) higherHyCell CITATION.cff: Geneformer higher in 4, UCE (Universal Cell Embeddings) higher in 6, 2 ties. Wu et al. 2025: Geneformer higher in 0, UCE (Universal Cell Embeddings) higher in 3, 1 tie. Deep-learning-based gene perturbation…: Geneformer higher in 2, UCE (Universal Cell Embeddings) higher in 0, 0 ties.HyCell CITATION.cffHyCell CITATION.cff (commit 0dcc533c)4 Geneformer higher42 tie26 UCE (Universal Cell Embeddings) higher6Wu et al. 2025Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13)1 tie13 UCE (Universal Cell Embeddings) higher3Deep-learning-based gene perturbation…Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines2 Geneformer higher2
Geneformer higherTieUCE (Universal Cell Embeddings) higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (3)
PaperRoleRanks Geneformer higherRanks UCE (Universal Cell Embeddings) higherTies
HyCell CITATION.cff (commit 0dcc533c) and 17 more records · RepositoryAgrees462
Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) · DOIAgrees031
Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines and 4 more records · DOIAgrees200
Comparisons in HyCell CITATION.cff (12)
ProtocolDatasetMetricGeneformerUCE (Universal Cell Embeddings)Higher
Annotation transfer with 15-nearest neighbours (HyCell report)Seven held-out atlases of the HyCell reportMacro F1 (Macro-F1 of a 15-nearest-neighbour classifier trained on the other batches, over the cell types of the held-out batch, averaged over held-out batches; per atlas or averaged over the seven atlases), fraction0.7820.814UCE (Universal Cell Embeddings)
Annotation transfer with logistic regression (HyCell report)Seven held-out atlases of the HyCell reportMacro F1 (Macro-F1 of a logistic-regression classifier trained on the other batches, over the cell types of the held-out batch, averaged over held-out batches; per atlas or averaged over the seven atlases), fraction0.8340.823Geneformer
scIB batch correction (HyCell report)Seven held-out atlases of the HyCell reportScib batch correction score (Mean of five scIB batch-correction metrics (batch silhouette, iLISI, kBET, graph connectivity and principal-component regression), per atlas or averaged over the seven atlases), unitless0.4650.432Geneformer
scIB biological conservation (HyCell report)Seven held-out atlases of the HyCell reportScib bio conservation score (Mean of seven scIB biological-conservation metrics (isolated-label score, NMI and ARI of k-means and of Leiden clusters against the cell types, cell-type silhouette and cLISI), per atlas or averaged over the seven atlases), unitless0.6060.662UCE (Universal Cell Embeddings)
Composite score S (HyCell report)Asian Immune Diversity Atlas, sample (HyCell report)Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless0.630.655UCE (Universal Cell Embeddings)
Composite score S (HyCell report)MultiVINE brain vasculature, healthy donors (HyCell report)Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless0.7560.758Tie
Composite score S (HyCell report)Cross-tissue immune atlas, stratified sample (HyCell report)Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless0.6340.63Tie
Composite score S (HyCell report)scIB human immune atlas, integer-count part (HyCell report)Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless0.6830.715UCE (Universal Cell Embeddings)
Composite score S (HyCell report)scIB human lung atlas, integer-count part (HyCell report)Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless0.6530.639Geneformer
Composite score S (HyCell report)scIB human pancreas atlas, integer-count part (HyCell report)Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless0.5790.668UCE (Universal Cell Embeddings)
Composite score S (HyCell report)PBMC from two 10x Genomics runs (HyCell report)Weighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless0.8160.793Geneformer
Composite score S (HyCell report)Seven held-out atlases of the HyCell reportWeighted composite score (S = 0.3 Bio + 0.2 Batch + 0.25 LR + 0.25 kNN, the authors' weighting, which gives the scIB total (0.6 Bio + 0.4 Batch) and annotation transfer equal weight; per atlas or averaged over the seven atlases), unitless0.6790.694UCE (Universal Cell Embeddings)
Comparisons in Wu et al. 2025 (4)
ProtocolDatasetMetricGeneformerUCE (Universal Cell Embeddings)Higher
Cross-atlas annotation transfer, HLCA to Tabula Sapiens, 14 shared cell types (Wu et al. 2025 Table S3)Tabula Sapiens (CELLxGENE), 120 leaf cell types after processingMacro F1 (test cells of the 14 shared cell types; averaged over 'all classes' (Methods 'Macro-F1'), not stated as the 14 shared types or every source-atlas type), fraction0.2390.283UCE (Universal Cell Embeddings)
Cross-atlas annotation transfer, HLCA to Tabula Sapiens, 14 shared cell types (Wu et al. 2025 Table S3)Tabula Sapiens (CELLxGENE), 120 leaf cell types after processingTop-1 accuracy (Accuracy@1 over cells of the 14 shared cell types), fraction0.7720.825UCE (Universal Cell Embeddings)
Cross-atlas annotation transfer, Tabula Sapiens to HLCA, 14 shared cell types (Wu et al. 2025 Table S2)HLCA core (CELLxGENE), 37 leaf cell types after processingMacro F1 (test cells of the 14 shared cell types; averaged over 'all classes' (Methods 'Macro-F1'), not stated as the 14 shared types or every source-atlas type), fraction0.0870.089UCE (Universal Cell Embeddings)
Cross-atlas annotation transfer, Tabula Sapiens to HLCA, 14 shared cell types (Wu et al. 2025 Table S2)HLCA core (CELLxGENE), 37 leaf cell types after processingTop-1 accuracy (Accuracy@1 over cells of the 14 shared cell types), fraction0.6650.669Tie
Comparisons in Deep-learning-based gene perturbation… (2)

What disagrees

studies
2
Compared
Geneformer, UCE (Universal Cell Embeddings)
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-fd487a3a29f37e0e · All contested results · How findings are made and followed up