Result

F1 (Simple Benchmark) of GPT-4o-mini, direct prompting, mean over prompting strategies (Song et al. 2025) on Virtual CRISPR Simple Benchmark

0.55 ± 0.04 F1 (Simple Benchmark)

Evidence concern: excluded from comparisons

Appendix A.2 describes the random baseline as predicting hit with probability equal to 'the overall proportion of actual hits in the specific benchmark dataset'. With one pooled rate (485 of 1,175 = 41.3%, Table 3) the expected PPV, sensitivity, F1 and FPR are all about 0.41 and specificity and NPV about 0.59. Table 8 prints PPV 0.47, sensitivity 0.46, F1 0.47, FPR 0.36, specificity 0.64 and NPV 0.63. With each screen's own hit rate (Table 3) the expected values are 0.47, 0.47, 0.47, 0.37, 0.63 and 0.63, which match. One draw at the pooled rate would need sensitivity about 2 standard errors above and FPR about 3 below expectation at once. The baseline therefore probably uses per-screen hit rates, which is information a screen designer would not have before the screen. The expected values were computed by the reviewer from Table 3 counts and are not stored as results. Treat the random row as a per-screen-prevalence classifier, not a matched random-selection comparator.Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? · Appendix A.2; Table 3; Table 1, row "Random Baseline"; Table 8, row "Random N/A N/A"

Methods

Tested configuration: GPT-4o-mini, direct prompting, mean over prompting strategies (Song et al. 2025)
Protocol: Virtual CRISPR Simple Benchmark: yes/no prediction of a screen hit
Dataset: Virtual CRISPR Simple Benchmark
Procedure: Virtual CRISPR Simple Benchmark: yes/no prediction of a screen hit
Evaluation: GPT-4o-mini (direct prompting, mean over strategies) on the Virtual CRISPR Simple Benchmark
Coverage: Not reported scored / Not reported eligible
Uncertainty: SD 0.04. Caption: mean and std. dev. across up to 10 combinations of CoT and few-shot prompting; the number of combinations per model is not printed
Evidence: Independent external evaluation · source checkedVirtual CRISPR: Can LLMs Predict CRISPR Screen Results? · Table 1, row "GPT-4o-mini", column "F1 Score"

A source-checked result verifies the numerical transcription, not every model or protocol detail. Evaluation metadata: source checked. Source checked does not mean independently reproduced.

Reproduction

Split: Fixed evaluation set of 1,175 queries; no training on it
Adaptation: Direct prompting, no fine-tuning
Scoring implementation: Binary Yes/No answer extracted from generated text; metrics over queries

No execution recipe has been verified for this exact configuration and evaluation. A benchmark's general instructions may use different inputs, splits or model settings.

Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

1 evidence row matching the loaded filters

Claims, original sources and review scope · Release 2026-10-23-5347695500ed
Property and statement	Original source and location	Review and provenance
Reported result 0.55 ± 0.04 Individual claims	Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? Original source ↗ Table 1, row "GPT-4o-mini", column "F1 Score" Version: Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30 Retrieved: 2026-10-22T13:57:26Z	source checked ["source-hash-verification","pdf-text-parse","independent-cell-check","rendered-source-visual-check"] · 2026-10-22 independent paper Source has a recorded evidence concern. Consult its source page before using the claim. Audit details Table 1 '±' is the standard deviation across prompting strategies (caption 'mean and std. dev.'); the ten Table 8 GPT-4o rows have mean F1 0.472 and FPR 0.223 and standard deviation about 0.06, which matches the Table 1 GPT-4o row. n per model is not printed. Field: `attributes.printed_value` Source artifact SHA-256: `6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c` Hash scope: pdftotext -layout text layer, parsed by data/omics/use-case-refresh-phenotype-20261022/extract/song2025virtualcrispr.py with row labels asserted Inspected artifact Extraction artifact SHA-256: `6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c` Extraction artifact

How to read the evidence

Sources and history

Release 2026-10-23-5347695500ed · Record review: source checked

1 source record and release history

Virtual CRISPR: Can LLMs Predict CRISPR Screen Results? · Original source · Proceedings of the 24th Workshop on Biomedical Language Processing (BioNLP 2025), pages 354-364, 1 August 2025; ACL Anthology 2025.bionlp-1.30

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: pheno-20261022-result-song2025-gpt-4o-mini-prompting-mean-simple-f1-score

metric: f1-score
metric qualifier: Simple Benchmark
metric direction: higher
unit: fraction
printed value: 0.55 ± 0.04
numeric value: 0.55
source locator: Table 1, row "GPT-4o-mini", column "F1 Score"
review: method: source-hash-verification; pdf-text-parse; independent-cell-check; rendered-source-visual-check; method note: Re-downloaded the PDF from the retrieval URL and matched its SHA-256; the regenerated pdftotext -layout text layer is byte-identical to the extractor's. Read every cell with a separate parser written for this review (review/song2025virtualcrispr_verify.py), with table headers, row labels and column positions asserted, so the column is checked as well as the row; the extractor script was not imported or run. Checked printed and numeric value, locator, metric, unit, direction, uncertainty, and the linked evaluation, configuration, Table 7 model version and protocol. Tables 1, 2 and 8 were also rendered from the PDF (pdftoppm) and compared by eye with the parsed rows, including the column under which each '±' value sits in Table 2. batch.py check-values returns row_not_found for these locators because Table 1 sits in the right-hand text column and Tables 2 and 8 print their captions below the rows.; reviewer: claude; reviewer note: Claude (Opus 5.5) review agent, independent of the extractor; no human review claimed; date: 2026-10-22; artifact sha256: 6c53ffec88444768fbb5c1a65031c096f07b3d9ac48de8163bc8f47d0d51205c; retrieval url: https://aclanthology.org/2025.bionlp-1.30.pdf; note: Table 1 '±' is the standard deviation across prompting strategies (caption 'mean and std. dev.'); the ten Table 8 GPT-4o rows have mean F1 0.472 and FPR 0.223 and standard deviation about 0.06, which matches the Table 1 GPT-4o row. n per model is not printed.
uncertainty: type: standard_deviation; value: 0.04; printed: ± 0.04; scope: across prompting-strategy combinations for this model; note: Caption: mean and std. dev. across up to 10 combinations of CoT and few-shot prompting; the number of combinations per model is not printed

Related records

evaluation: GPT-4o-mini (direct prompting, mean over strategies) on the Virtual CRISPR Simple Benchmark

Suggest a correction