AssayBench: An Assay-Level Virtual Cell Benchmark for LLMs and Agents

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 5 models
• 7 methods
• 16 configurations
• 1 protocol
• 3 datasets
• 48 evaluations
• 144 results
• 4 evidence claims
• 1 use case

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• Evidence summary: therapeutic target validation: Summary Three sources compare target-nomination methods against measured CRISPR screen outcomes, and all three are retrospective replays of published screens: a method picks genes and the screen's existing measurement is looked up. Non-hits are genes the screen assayed that did not meet its criterion; genes a screen never assayed are outside the candidate pool or, in AssayBench's headline metric, removed before scoring, so these results say nothing about an untested target. Roohani et al. 2025 (ICLR) were run by the developers of the agent they test, two of whom are authors of two of the screens, and they state that patent applications have been filed on the findings. They report the fraction of each screen's hits found after five rounds of 128 genes. With Claude 3.5 Sonnet the agent reached 0.095 on the interferon-gamma screen, 0.104 on interleukin-2, 0.130 on CAR-T proliferation, 0.326 on lysosomal choline recycling, 0.042 on T-cell inhibition resistance and 0.066 on tau level, against random selection at 0.037, 0.031, 0.033, 0.160, 0.036 and 0.034. Conventional baselines were close on some screens: Coreset reached 0.102 on interleukin-2 and 0.047 on T-cell inhibition resistance, above the agent's 0.042 there. Gupta et al. 2025 (Findings of EMNLP), independent of the agent's authors, replayed five of the same screens and report cumulative hits. Their re-run of the Claude 3.5 Sonnet agent found fewer hits than the original numbers on three of four shared screens, for example 31.6 against 60.72 on tau level. Giving the agent randomly permuted feedback changed little: the permuted variant scored at least as high on three of five screens with each of the three backbones tested, for example 51 against 44 on interferon-gamma with Llama-3.1-8B, and the authors conclude that the models they tested do not perform in-context experimental design. Gaussian process optimisation over the same embeddings reached 147.8 hits on interleukin-2 against the agent's 39.4 with Llama-3.1-8B, from 654 ground-truth hits; the source prints identical Gaussian process values under both backbones. De Brouwer et al. 2026, a preprint that is not peer reviewed, asked 16 systems to rank 100 genes for each of 571 benchmark entries, each a described screen. On the 334 test entries the best system was an evolved ensemble of language models at 0.1631 adjusted NDCG@100 and the best single model, Gemini 3 Pro, 0.1570, against 0.1334 for a baseline ranking genes by how often they are hits in training screens of the same phenotype and 0.2918 for a retrieval upper bound that uses the answer. Most systems scored lower on the 19 entries published after the models' training cutoff, the ensemble falling to 0.1088, which that source reads as partly prior exposure to the literature. None of these sources tests a prospective validation batch, a target's usefulness, dependency, tractability or therapeutic window, or whether modulating a target benefits patients. (tgtval-20261009-source-roohani2025: Table 1, sections 2 and 4.1, conflicts of interest; tgtval-20261009-source-gupta2025: Tables 1 and 2, abstract, section 5; tgtval-20261009-source-debrouwer2026: Tables 2 and 3, sections 2 to 5)
• GPT-OSS-120B: description: Description Open-weight reasoning model from OpenAI with a mixture-of-experts Transformer architecture, trained by large-scale distillation and reinforcement learning. (qa-descriptions-20261011-b-source-gpt-oss: abstract)
• label_semantics: tgtval-20261009-data-assaybench-test: Label semantics A hit is a gene that satisfies the significance criterion of its own screen. Screens where all tested genes are significant, or with missing criteria, were removed. Relevance scores are percentile ranks of the screen's own metrics combined by geometric mean, with non-hit genes set to zero; screens whose relevance ranking disagreed with the original hit labels at ROC-AUC below 0.95 were excluded. Phenotype descriptions and effect directions were extracted by an LLM-assisted curation step. (Sections 2.1 and 2.2)
• Relevance of tgtval-20261009-protocol-debrouwer2026-screen-gene-ranking to "Which targets should I test to change a defined disease-relevant phenotype?": Assessed by AssayBench: rank 100 candidate genes for a described CRISPR screen (tgtval-20261009-source-debrouwer2026: Sections 2 to 4; Tables 1 to 3)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.