Biology-in-the-loop: Amortized Adaptive Hit Discovery in CRISPR Screens

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 10 models
• 12 methods
• 53 configurations
• 1 protocol
• 1 dataset subset
• 1 baseline
• 53 evaluations
• 530 results
• 7 evidence claims

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• composition (assaybench-loop-composition): Composition AssayBench-Loop comprises 1,389 CRISPR screens across five phenotype categories, split temporally into 1,349 training (before 2021), 20 validation (2021) and 20 test (after 2021) screens; median test-to-training hit-set Jaccard similarity 0.064. (Section 2.1; Figure 2B caption)
• Evidence summary: Select perturbations for a defined cellular response: Summary The evidence comes from one prospective challenge and two retrospective replays, which measure different things. In the Cancer Immunotherapy Data Science Challenge, scored prospectively in mouse T cells, the 20 original Challenge 2 submissions had a top-k overlap AUC of 0.452 to 0.566, with 17 of the 20 between 0.50 and 0.57; a random ordering is expected to score 0.5. That score is not a ROC AUC, and the genes it scores were drawn from the leading Challenge 1 nominations. In the follow-up, two nominated targets exceeded the Screen 1 targets on the immune checkpoint blockade objective; the source reports 61 nominations, 57 selected targets, 59 screen-2 genes and 50 perturbations passing quality control without reconciling them, so no success fraction follows. AssayBench-Loop replays 20 genome-wide CRISPR screens published after 2021, with each policy choosing 10 rounds of 100 genes. One random draw found 4.9% of each screen's hits on average (enrichment 1.04 over the random expectation). Language models used directly as the policy, with hit labels, reached enrichment of 1.39 to 4.81, BPMF 4.49, and the authors' handoffs from a language model to their AssayFormer up to 5.67, with the Gemini-3.1-Pro handoff finding 27.7% of hits. Many recovered hits are common-essential genes: 30.1% for that handoff, 18.6% for the random draw and 99.1% for a baseline that picks earlier screens' hits. Test screens were kept only if the best language model already passed a performance floor on them, and the paper does not say whether the models saw the screens' publications in training. The authors ran every row, and no uncertainty is printed. For two-gene combinations, NAIAD's replay of four viability screens counted how many of each screen's 200 strongest pairs the model ranked in its top 200 after four rounds: 143.0, 141.7, 99.7 and 150.0 when it acquired the pairs with the largest predicted effect, against 93.3, 70.3, 38.7 and 81.3 when it acquired pairs uniformly at random (means of three replicates). That count includes pairs the model predicted but never measured, and the uniform arm still uses the model's ranking, so it is not the yield of testing random pairs. The challenge's submissions scored close to a random ordering, while in both replays the leading model-guided policies scored above the random or uniform arm; the two differ in endpoint, candidate set, setting and who ran them, so their results cannot be compared directly. No source compares a model shortlist with random selection prospectively at the same budget. (ucc-research-source-challenge-supp-s2: Supplementary Table S2, media-3.xlsx, sheet 'Challenge 1 & 2', column D rows 2 to 24; ucc-research-source-challenge-paper: Abstract and Results Figure 5; ucpheno-20261022-source-edwards2026: Table 1, rows 'Random', 'GPT-5.6 Sol', 'BPMF [41]', 'Gemini-3.1-Pro → AssayFormer', 'GPT-5.6 Sol → AssayFormer' and 'Prior hit baseline', columns 'EF', 'FH (%)' and '%ess (%)'; Methods 4.2 and Appendix A (test-screen criteria); Supplementary Table 4, Base LLMs group, column 'EF'; pheno-20261022-qin2025-source-pmlr: Table 4, rows 'NAIAD MPE' and 'NAIAD Uniform')
• handoff (assayloop-handoff): Handoff A large language model acts as the acquisition policy for the first k rounds and AssayFormer for the remaining T - k rounds; k = 3 was selected on the validation set. (Section 2.2; Methods 4.6)
• post_training (assayllm-training): Post training AssayLLM is Qwen3.6-27B supervised fine-tuned on GLM-5.1 acquisition trajectories from the training screens and then trained with GRPO; the authors report EF rising from 2.65 (base) to 3.56 (SFT) and 3.69 (SFT + GRPO). (Section 2.7; Figure 6 caption; Methods 4.7)
• prospective_validation (prospective-validation): Prospective validation The authors state that AssayBench-Loop is a retrospective benchmark and that the policies remain to be validated prospectively in a live experimental loop. (Section 3 (Discussion), limitations paragraph)
• Relevance of AssayBench-Loop adaptive hit discovery: 10 rounds of 100 genes on 20 held-out CRISPR screens to "Select perturbations for a defined cellular response": Assessed by AssayBench-Loop adaptive hit discovery: 10 rounds of 100 genes on 20 held-out CRISPR screens (ucpheno-20261022-source-edwards2026: Table 1 and caption; Supplementary Table 4; Methods 4.2 and 4.3; Appendix H; Discussion (limitations))
• training (assayformer-training): Training Gene embeddings are initialised by BPMF on the training hit matrix; the policy is trained with binary cross-entropy on simulated partial histories, then fine-tuned with GRPO on full 10-round rollouts using a context-delta reward relative to a context-free reference policy. (Section 2.2; Figure 3C caption; Methods 4.5)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.