rewirebio.iobenchmarks
Evidence and investigation

Contested results

Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.

These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.

A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.

How the check works, and how each follow-up is recorded: discrepancies between studies.

In this release

  • 192open
  • 6settled
  • 7resolved
  • 3medium severity
  • 202low severity

Release 2026-11-21-c8bea25e5abb · 205 findings in all · rules version 5

Findings

Status: All · Open · Settled · Resolved

Severity: All · Medium · Low

State: All · Not yet reviewed · Under investigation · Explained · Confirmed error · Not a real disagreement

Showing findings that involve results in Assess models for genetic perturbation experiments, from the study that disagrees or from the agreeing group. All use cases

4 findings match, most severe first.

Geneformer and scFoundation: Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines ranks Geneformer above scFoundation in 2 of 2 comparisons; 2 of 2 other studies rank scFoundation above Geneformer.

2 papers rank scFoundation above Geneformer in 13 of their 16 comparisons and Geneformer higher in 3. Deep-learning-based gene perturbation… ranks Geneformer above scFoundation in all 2 of its comparisons.

Comparisons in which each paper scores scFoundation or Geneformer higher2 other papers: scFoundation higher in 13, Geneformer higher in 3, 0 ties. Deep-learning-based gene perturbation… (disagrees): scFoundation higher in 0, Geneformer higher in 2, 0 ties.2 other papers13 scFoundation higher133 Geneformer higher3Disagrees: Deep-learning-based gene perturbation…Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines2 Geneformer higher2
scFoundation higherTieGeneformer higher
Compared
Geneformer, scFoundation

Not reviewed yet.

scFoundation and scGPT: Csendes et al. 2025, Supplementary Material 4 (Supplementary Tables 1-3) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Csendes et al. 2025, Supplementary Material 4 (Supplementary Tables 1-3) ranks scGPT above scFoundation in 15 of 16 comparisons; 2 of 2 other studies rank scFoundation above scGPT.

3 papers rank scFoundation above scGPT in 18 of their 25 comparisons and scGPT higher in 5; 2 are ties. Csendes et al. 2025 ranks scGPT above scFoundation in 15 of its 20 comparisons and scFoundation higher in 1; 4 are ties.

Comparisons in which each paper scores scFoundation or scGPT higher3 other papers: scFoundation higher in 18, scGPT higher in 5, 2 ties. Csendes et al. 2025 (disagrees): scFoundation higher in 1, scGPT higher in 15, 4 ties.3 other papers18 scFoundation higher182 tie5 scGPT higher5Disagrees: Csendes et al. 2025Csendes et al. 2025, Supplementary Material 4 (Supplementary Tables 1-3)1 scFoundation higher4 tie415 scGPT higher15
scFoundation higherTiescGPT higher
Compared
scFoundation, scGPT

Not reviewed yet.

Geneformer and scGPT: Kedzierska et al. 2025, Additional file 1 (Supplementary Tables S1-S7, xlsx) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Kedzierska et al. 2025, Additional file 1 (Supplementary Tables S1-S7, xlsx) ranks scGPT above Geneformer in 83 of 89 comparisons; 2 of 2 other studies rank Geneformer above scGPT.

4 papers rank Geneformer above scGPT in 10 of their 19 comparisons and scGPT higher in 5; 4 are ties. Of those papers, Parameter-Efficient Fine-Tuning… ranks scGPT higher on its own (1 of 1). Kedzierska et al. 2025 ranks scGPT above Geneformer in 83 of its 97 comparisons and Geneformer higher in 6; 8 are ties.

Comparisons in which each paper scores Geneformer or scGPT higher4 other papers: Geneformer higher in 10, scGPT higher in 5, 4 ties. Kedzierska et al. 2025 (disagrees): Geneformer higher in 6, scGPT higher in 83, 8 ties.4 other papers10 Geneformer higher104 tie45 scGPT higher5Disagrees: Kedzierska et al. 2025Kedzierska et al. 2025, Additional file 1 (Supplementary Tables S1-S7, xlsx)6 Geneformer higher8 tie83 scGPT higher83
Geneformer higherTiescGPT higher
Compared
Geneformer, scGPT

Not reviewed yet.

Geneformer and UCE (Universal Cell Embeddings): Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines and Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines ranks Geneformer above UCE (Universal Cell Embeddings) in 2 of 2 comparisons; Wu et al. 2025, Additional file 3 (Supplementary Tables S1-S13) ranks UCE (Universal Cell Embeddings) above Geneformer in 3 of 3 comparisons. No other study gives a verdict, so neither is the outlier.

3 papers rank Geneformer above UCE (Universal Cell Embeddings) in 6 of their 18 comparisons and UCE (Universal Cell Embeddings) higher in 9; 3 are ties. Of those papers, Wu et al. 2025 ranks UCE (Universal Cell Embeddings) higher on its own (3 of 4).

Comparisons in which each paper scores Geneformer or UCE (Universal Cell Embeddings) higher3 other papers: Geneformer higher in 6, UCE (Universal Cell Embeddings) higher in 9, 3 ties.3 other papers6 Geneformer higher63 tie39 UCE (Universal Cell Embeddings) higher9
Geneformer higherTieUCE (Universal Cell Embeddings) higher
Compared
Geneformer, UCE (Universal Cell Embeddings)

Not reviewed yet.