AssayBench: rank 100 candidate genes for a described CRISPR screen

Stable ID: tgtval-20261009-protocol-debrouwer2026-screen-gene-ranking

areas

cells-tissues

contexts

research

protocol

For each benchmark entry the system is given the phenotype, cell line, cell type, library and perturbation modality, treatment and duration, plus the ranking objective, and returns 100 ranked gene symbols. Metrics: AnDCG@100 (condensed NDCG adjusted so 0 is a random ranking for that screen), Precision@100, and dFDR@100.

version

AssayBench, arXiv:2605.10876v1, sections 2 to 4

metric

adjusted-ndcg-at-100

metric direction

higher

unit

unitless

metric definition

AnDCG@100 removes unassayed genes from the ranked list, computes NDCG@100 against percentile-rank relevance, and rescales by the screen's analytic random baseline. Precision@100 counts hits among the top 100 predictions and divides by min(100, positive-relevance genes), so for screens with fewer than 100 hits it is the fraction of the screen's hits recovered. dFDR@100 is the share of the top 100 scored genes that are significant in the opposite direction.

limitations

AnDCG@100 removes genes the screen did not assay before scoring, so for the headline metric untested genes are not negatives. The printed Precision@100 and dFDR@100 formulas sum over the top 100 predictions without that step, so an unassayed gene adds nothing to Precision@100 and counts in the dFDR@100 denominator; the text calls them measures over in-screen predictions, so whether they were condensed is unclear.; Hits follow each screen's own significance criterion, so the phenotype and threshold vary between entries.; Retrospective replay of published screens; no new experiment was run.; Phenotype descriptions and effect directions come from an LLM-assisted curation step over BioGRID metadata.; The post-cutoff cohort has 19 screens against 334 in the test cohort. Most systems score lower on it, which the source reads as partly prior exposure to the literature; the trained gene-relevance predictor (0.0660 against 0.0565) and Qwen3.5-2B (0.0324 against 0.0284) score higher.; Precision@100 divides by min(100, number of positive-relevance genes), so it is not precision over a fixed 100 predictions when a screen has fewer than 100 hits.; The gene-frequency baseline is competitive mainly through fitness and viability screens, where frequently recurring essential genes give a phenotype-level prior (section 5.4).

source locator

Sections 2.4, 3 and 4; Tables 1 to 3

• uses data: AssayBench validation cohort of human CRISPR screens
• uses data: AssayBench test cohort of human CRISPR screens
• uses data: AssayBench post-cutoff cohort of human CRISPR screens
• assessment: Biomni A1 (Claude 4) on the AssayBench post-cutoff cohort
• assessment: Biomni A1 (Claude 4) on the AssayBench test cohort
• assessment: Biomni A1 (Claude 4) on the AssayBench validation cohort
• assessment: C2S (Gemma-2B) on the AssayBench post-cutoff cohort
• assessment: C2S (Gemma-2B) on the AssayBench test cohort
• assessment: C2S (Gemma-2B) on the AssayBench validation cohort
• assessment: Embedding kNN on the AssayBench post-cutoff cohort
• assessment: Embedding kNN on the AssayBench test cohort
• assessment: Embedding kNN on the AssayBench validation cohort
• assessment: Gemini 3 Flash (GEPA) on the AssayBench post-cutoff cohort
• assessment: Gemini 3 Flash (GEPA) on the AssayBench test cohort
• assessment: Gemini 3 Flash (GEPA) on the AssayBench validation cohort
• assessment: Gemini 3 Flash on the AssayBench post-cutoff cohort
• assessment: Gemini 3 Flash on the AssayBench test cohort
• assessment: Gemini 3 Flash on the AssayBench validation cohort
• assessment: Gemini 3 Pro (Few-shot) on the AssayBench post-cutoff cohort
• assessment: Gemini 3 Pro (Few-shot) on the AssayBench test cohort
• assessment: Gemini 3 Pro (Few-shot) on the AssayBench validation cohort
• assessment: Gemini 3 Pro on the AssayBench post-cutoff cohort
• assessment: Gemini 3 Pro on the AssayBench test cohort
• assessment: Gemini 3 Pro on the AssayBench validation cohort
• assessment: Gene-frequency (by phenotype) on the AssayBench post-cutoff cohort
• assessment: Gene-frequency (by phenotype) on the AssayBench test cohort
• assessment: Gene-frequency (by phenotype) on the AssayBench validation cohort
• assessment: Gene-relevance predictor on the AssayBench post-cutoff cohort
• assessment: Gene-relevance predictor on the AssayBench test cohort
• assessment: Gene-relevance predictor on the AssayBench validation cohort
• assessment: GPT-5.4 on the AssayBench post-cutoff cohort
• assessment: GPT-5.4 on the AssayBench test cohort
• assessment: GPT-5.4 on the AssayBench validation cohort
• assessment: GPT-OSS-120B on the AssayBench post-cutoff cohort
• assessment: GPT-OSS-120B (SFT + GRPO) on the AssayBench post-cutoff cohort
• assessment: GPT-OSS-120B (SFT + GRPO) on the AssayBench test cohort
• assessment: GPT-OSS-120B (SFT + GRPO) on the AssayBench validation cohort
• assessment: GPT-OSS-120B (SFT) on the AssayBench post-cutoff cohort
• assessment: GPT-OSS-120B (SFT) on the AssayBench test cohort
• assessment: GPT-OSS-120B (SFT) on the AssayBench validation cohort
• assessment: GPT-OSS-120B on the AssayBench test cohort
• assessment: GPT-OSS-120B on the AssayBench validation cohort
• assessment: LLM Ensemble on the AssayBench post-cutoff cohort
• assessment: LLM Ensemble on the AssayBench test cohort
• assessment: LLM Ensemble on the AssayBench validation cohort
• assessment: Oracle kNN on the AssayBench post-cutoff cohort
• assessment: Oracle kNN on the AssayBench test cohort
• assessment: Oracle kNN on the AssayBench validation cohort
• assessment: Qwen3.5-2B on the AssayBench post-cutoff cohort
• assessment: Qwen3.5-2B on the AssayBench test cohort
• assessment: Qwen3.5-2B on the AssayBench validation cohort
• assessed by: Select therapeutic targets for validation