Result

False positive rate (Difficult Benchmark) of Virtual CRISPR embedding classifier, raw-text embeddings (Song et al. 2025) on Virtual CRISPR Difficult Benchmark

0.15 False positive rate (Difficult Benchmark)

Evidence concern: excluded from comparisons

Appendix A.2 describes the random baseline as predicting hit with probability equal to 'the overall proportion of actual hits in the specific benchmark dataset'. With one pooled rate (485 of 1,175 = 41.3%, Table 3) the expected PPV, sensitivity, F1 and FPR are all about 0.41 and specificity and NPV about 0.59. Table 8 prints PPV 0.47, sensitivity 0.46, F1 0.47, FPR 0.36, specificity 0.64 and NPV 0.63. With each screen's own hit rate (Table 3) the expected values are 0.47, 0.47, 0.47, 0.37, 0.63 and 0.63, which match. One draw at the pooled rate would need sensitivity about 2 standard errors above and FPR about 3 below expectation at once. The baseline therefore probably uses per-screen hit rates, which is information a screen designer would not have before the screen. The expected values were computed by the reviewer from Table 3 counts and are not stored as results. Treat the random row as a per-screen-prevalence classifier, not a matched random-selection comparator.Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? · Appendix A.2; Table 3; Table 1, row "Random Baseline"; Table 8, row "Random N/A N/A"

Methods

Tested configuration: Virtual CRISPR embedding classifier, raw-text embeddings (Song et al. 2025)
Protocol: Virtual CRISPR Difficult Benchmark: hit prediction for complex phenotypes
Dataset: Virtual CRISPR Difficult Benchmark
Procedure: Virtual CRISPR Difficult Benchmark: hit prediction for complex phenotypes
Evaluation: Raw Emb. on the Virtual CRISPR Difficult Benchmark
Coverage: Not reported scored / Not reported eligible
Uncertainty: Not reported by the source
Evidence: Author-reported evaluation · source checkedVirtual CRISPR: Can LLMs Predict CRISPR Screen Results? · Table 2, row "Raw Emb.", column "FPR"

A source-checked result verifies the numerical transcription, not every model or protocol detail. Evaluation metadata: source checked. Source checked does not mean independently reproduced.

Reproduction

Split: Fixed evaluation set of 1,814 queries; no data from BioGRID-ORCS
Adaptation: Supervised MLP on LLM embeddings, trained on BioGRID-ORCS
Scoring implementation: AUROC and AUPRC from predicted probabilities; other metrics after binarising with Youden's J statistic

No execution recipe has been verified for this exact configuration and evaluation. A benchmark's general instructions may use different inputs, splits or model settings.

Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

1 evidence row matching the loaded filters

Claims, original sources and review scope · Release 2026-10-23-5347695500ed
Property and statement	Original source and location	Review and provenance
Reported result 0.15 Individual claims	Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? Original source ↗ Table 2, row "Raw Emb.", column "FPR" Version: Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30 Retrieved: 2026-10-22T13:57:26Z	source checked ["source-hash-verification","pdf-text-parse","independent-cell-check","rendered-source-visual-check"] · 2026-10-22 author reported Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Threshold-dependent: Youden's J threshold, selection data not stated. Field: `attributes.printed_value` Source artifact SHA-256: `6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c` Hash scope: pdftotext -layout text layer, parsed by data/omics/use-case-refresh-phenotype-20261022/extract/song2025virtualcrispr.py with row labels asserted Inspected artifact Extraction artifact SHA-256: `6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c` Extraction artifact

How to read the evidence

Sources and history

Release 2026-10-23-5347695500ed · Record review: source checked

1 source record and release history

Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? · Original source · Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: pheno-20261022-result-song2025-embedding-classifier-raw-difficult-false-positive-rate

metric: false-positive-rate
metric qualifier: Difficult Benchmark
metric direction: lower
unit: fraction
printed value: 0.15
numeric value: 0.15
source locator: Table 2, row "Raw Emb.", column "FPR"
review: method: source-hash-verification; pdf-text-parse; independent-cell-check; rendered-source-visual-check; method note: Re-downloaded the PDF from the retrieval URL and matched its SHA-256; the regenerated pdftotext -layout text layer is byte-identical to the extractor's. Read every cell with a separate parser written for this review (review/song2025virtualcrispr_verify.py), with table headers, row labels and column positions asserted, so the column is checked as well as the row; the extractor script was not imported or run. Checked printed and numeric value, locator, metric, unit, direction, uncertainty, and the linked evaluation, configuration, Table 7 model version and protocol. Tables 1, 2 and 8 were also rendered from the PDF (pdftoppm) and compared by eye with the parsed rows, including the column under which each '±' value sits in Table 2. batch.py check-values returns row_not_found for these locators because Table 1 sits in the right-hand text column and Tables 2 and 8 print their captions below the rows.; reviewer: claude; reviewer note: Claude (Opus 5.5) review agent, independent of the extractor; no human review claimed; date: 2026-10-22; artifact sha256: 6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c; retrieval url: https://aclanthology.org/2025.bionlp-1.30.pdf; note: Threshold-dependent: Youden's J threshold, selection data not stated.
missing metadata: uncertainty: reason: unreported

Related records

evaluation: Raw Emb. on the Virtual CRISPR Difficult Benchmark

Suggest a correction