Evidence claim

leakage_control: pheno-20261022-data-song2025-difficult

Evidence concern: excluded from comparisons

Appendix A.2 describes the random baseline as predicting hit with probability equal to 'the overall proportion of actual hits in the specific benchmark dataset'. With one pooled rate (485 of 1,175 = 41.3%, Table 3) the expected PPV, sensitivity, F1 and FPR are all about 0.41 and specificity and NPV about 0.59. Table 8 prints PPV 0.47, sensitivity 0.46, F1 0.47, FPR 0.36, specificity 0.64 and NPV 0.63. With each screen's own hit rate (Table 3) the expected values are 0.47, 0.47, 0.47, 0.37, 0.63 and 0.63, which match. One draw at the pooled rate would need sensitivity about 2 standard errors above and FPR about 3 below expectation at once. The baseline therefore probably uses per-screen hit rates, which is information a screen designer would not have before the screen. The expected values were computed by the reviewer from Table 3 counts and are not stored as results. Treat the random row as a per-screen-prevalence classifier, not a matched random-selection comparator.Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? · Appendix A.2; Table 3; Table 1, row "Random Baseline"; Table 8, row "Random N/A N/A"

Leakage control of Virtual CRISPR Difficult Benchmark

The Difficult Benchmark is built from two publications released in late 2024, selected because they post-date the training cutoffs of all evaluated LLMs, and contains no data from BioGRID-ORCS.

Descriptive fact transcribed from the pinned source.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

6 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-23-5347695500ed
Property and statement	Original source and location	Review and provenance
Claimed field leakage_control Context-only references	Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? Original source ↗ Section 2.1 item (2); Section 3 last sentence; Limitations Version: Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30 Retrieved: 2026-10-22T13:57:26Z	not individually reviewed No individual claim review recorded Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.field` Source artifact SHA-256: `6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c` Hash scope: pdftotext -layout text layer, parsed by data/omics/use-case-refresh-phenotype-20261022/extract/song2025virtualcrispr.py with row labels asserted Inspected artifact
Source location Section 2.1 item (2); Section 3 last sentence; Limitations Context-only references	Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? Original source ↗ Section 2.1 item (2); Section 3 last sentence; Limitations Version: Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30 Retrieved: 2026-10-22T13:57:26Z	not individually reviewed No individual claim review recorded Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.source_locator` Source artifact SHA-256: `6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c` Hash scope: pdftotext -layout text layer, parsed by data/omics/use-case-refresh-phenotype-20261022/extract/song2025virtualcrispr.py with row labels asserted Inspected artifact
Claimed value The Difficult Benchmark is built from two publications released in late 2024, selected because they post-date the training cutoffs of all evaluated LLMs, and contains no data from BioGRID-ORCS. Context-only references	Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? Original source ↗ Section 2.1 item (2); Section 3 last sentence; Limitations Version: Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30 Retrieved: 2026-10-22T13:57:26Z	not individually reviewed No individual claim review recorded Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `attributes.value` Source artifact SHA-256: `6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c` Hash scope: pdftotext -layout text layer, parsed by data/omics/use-case-refresh-phenotype-20261022/extract/song2025virtualcrispr.py with row labels asserted Inspected artifact
Description Descriptive fact transcribed from the pinned source. Context-only references	Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? Original source ↗ Section 2.1 item (2); Section 3 last sentence; Limitations Version: Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30 Retrieved: 2026-10-22T13:57:26Z	not individually reviewed No individual claim review recorded Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `description` Source artifact SHA-256: `6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c` Hash scope: pdftotext -layout text layer, parsed by data/omics/use-case-refresh-phenotype-20261022/extract/song2025virtualcrispr.py with row labels asserted Inspected artifact
Relationship: subject pheno-20261022-data-song2025-difficult Context-only references	Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? Original source ↗ Section 2.1 item (2); Section 3 last sentence; Limitations Version: Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30 Retrieved: 2026-10-22T13:57:26Z	not individually reviewed No individual claim review recorded Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `links:subject:pheno-20261022-data-song2025-difficult` Source artifact SHA-256: `6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c` Hash scope: pdftotext -layout text layer, parsed by data/omics/use-case-refresh-phenotype-20261022/extract/song2025virtualcrispr.py with row labels asserted Inspected artifact
Name leakage_control: pheno-20261022-data-song2025-difficult Context-only references	Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? Original source ↗ Section 2.1 item (2); Section 3 last sentence; Limitations Version: Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30 Retrieved: 2026-10-22T13:57:26Z	not individually reviewed No individual claim review recorded Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Field: `name` Source artifact SHA-256: `6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c` Hash scope: pdftotext -layout text layer, parsed by data/omics/use-case-refresh-phenotype-20261022/extract/song2025virtualcrispr.py with row labels asserted Inspected artifact

How to read the evidence

Sources and history

Release 2026-10-23-5347695500ed · Record review: source checked

1 source record and release history

Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? · Original source · Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: pheno-20261022-claim-song2025-leakage-control

field: leakage_control
value: The Difficult Benchmark is built from two publications released in late 2024, selected because they post-date the training cutoffs of all evaluated LLMs, and contains no data from BioGRID-ORCS.
source locator: Section 2.1 item (2); Section 3 last sentence; Limitations
review: method: source-hash-verification; ai-assisted-source-review; method note: Read the cited sections of the pinned PDF text layer and checked the claim's wording against them.; reviewer: claude; reviewer note: Claude (Opus 5.5) review agent, independent of the extractor; no human review claimed; date: 2026-10-22; artifact sha256: 6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c; retrieval url: https://aclanthology.org/2025.bionlp-1.30.pdf; note: Matches Section 2.1 item (2) and Section 3. The paper asserts post-cutoff selection but prints no model cutoff dates.

Related records

subject: Virtual CRISPR Difficult Benchmark

Suggest a correction