Result

F1 (SNVs and indels, HG008 all-benchmark region) of Majority of Mutect2, Strelka2 and DeepSomatic (at least two callers) (Hasan 2026) on NIST/GIAB HG008 tumour-normal WGS (HG008-T pancreatic cancer cell line, matched normal), HG008-T draft somatic small-variant benchmark v0.3

0.794 F1 (SNVs and indels, HG008 all-benchmark region)

Methods

Tested configuration: Majority of Mutect2, Strelka2 and DeepSomatic (at least two callers) (Hasan 2026)
Protocol: HG008 all-benchmark region (sensitivity analysis), SNVs and indels combined, Mutect2, Strelka2, DeepSomatic and ensembles (Hasan 2026 Table 4)
Dataset: NIST/GIAB HG008 tumour-normal WGS (HG008-T pancreatic cancer cell line, matched normal), HG008-T draft somatic small-variant benchmark v0.3
Procedure: HG008 all-benchmark region (sensitivity analysis), SNVs and indels combined, Mutect2, Strelka2, DeepSomatic and ensembles (Hasan 2026 Table 4)
Evaluation: Majority on HG008 all-benchmark region (sensitivity analysis), SNVs and indels combined (Hasan 2026)
Coverage: Not reported scored / Not reported eligible
Uncertainty: 95% CI 0.7875 to 0.8003. Genomic block bootstrap over 5-Mb blocks, linear percentile interval; prespecified, but executed after the primary external evaluation; spatial variation within the benchmark region, not between tumours
Evidence: Author-reported evaluation · source checkedProspectively frozen external benchmarking reveals benchmark-context dependence in somatic variant calling; AmbioSpire P003/T001 publication source data and reproducibility package v1.0.0 (Zenodo) · Zenodo tables/Table4_HG008_sensitivity_all_region.tsv, row 'Majority', column 'F1'

A source-checked result verifies the numerical transcription, not every model or protocol detail. Evaluation metadata: source checked. Source checked does not mean independently reproduced.

Reproduction

Split: External test; procedure frozen on HCC1395 before HG008 truth access
Adaptation: Not reported
Scoring implementation: Exact normalised CHROM/POS/REF/ALT match of PASS calls within the region

No execution recipe has been verified for this exact configuration and evaluation. A benchmark's general instructions may use different inputs, splits or model settings.

Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

2 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-22-1b7cf027e4f6
Property and statement	Original source and location	Review and provenance
Reported result 0.793961 Individual claims	Prospectively frozen external benchmarking reveals benchmark-context dependence in somatic variant calling Original source ↗ Zenodo tables/Table4_HG008_sensitivity_all_region.tsv, row 'Majority', column 'F1' Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Research Square preprint rs-11210923 version 1, posted 2026-10-05; manuscript PDF (16 pages) Retrieved: 2026-10-11T21:39:12Z	source checked ["source-hash-verification","deterministic-table-parse","independent-cell-check"] · 2026-10-11 author reported Audit details Extraction: Read tables/Table4_HG008_sensitivity_all_region.tsv from the pinned Zenodo zip with extract/extract_hg008.py; member SHA-256 checked against the archive's SHA256SUMS; method and class row labels asserted. Zenodo tables/Table4_HG008_sensitivity_all_region.tsv, row 'Majority', column 'F1' Independent review 2026-10-11: value, locator, metric, unit, direction and linked configuration and protocol checked; corrections from the review applied by the extractor as listed in review.md (docs/reviews/use-cases/tumour-dna-somatic-variant-detection-2026-10-11.md). Field: `attributes.printed_value` Source artifact SHA-256: `358502e2394c1f90214889c40aae8b29d2836cddbcf55460688d7fda051dc1c3` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact Extraction artifact SHA-256: `629921b1198f4a5290d191857d13f98e54a1d132d722caf0ac83805f7e24a40d` Extraction artifact
Reported result 0.793961 Individual claims	AmbioSpire P003/T001 publication source data and reproducibility package v1.0.0 (Zenodo) Original source ↗ Zenodo tables/Table4_HG008_sensitivity_all_region.tsv, row 'Majority', column 'F1' Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Zenodo record 22998005, version 1.0.0, published 2026-09-27; file P003-T001-publication-data-v1.0.0-zenodo-final.zip (1,934,562 bytes, md5 a1fdab6108197602f07f72db77d9f996) Retrieved: 2026-10-11T21:39:34Z	source checked ["source-hash-verification","deterministic-table-parse","independent-cell-check"] · 2026-10-11 author reported Audit details Extraction: Read tables/Table4_HG008_sensitivity_all_region.tsv from the pinned Zenodo zip with extract/extract_hg008.py; member SHA-256 checked against the archive's SHA256SUMS; method and class row labels asserted. Zenodo tables/Table4_HG008_sensitivity_all_region.tsv, row 'Majority', column 'F1' Independent review 2026-10-11: value, locator, metric, unit, direction and linked configuration and protocol checked; corrections from the review applied by the extractor as listed in review.md (docs/reviews/use-cases/tumour-dna-somatic-variant-detection-2026-10-11.md). Field: `attributes.printed_value` Source artifact SHA-256: `629921b1198f4a5290d191857d13f98e54a1d132d722caf0ac83805f7e24a40d` Hash scope: SHA-256 of the downloaded zip. Member SHA-256 values for the extracted tables: tables/Table1_HG008_primary_metrics.tsv df3ac2809164be08676c9a010d3c7324ca08998a4b3964d5fb0c7a2201445fd6; tables/Table2_HG008_variant_class_metrics.tsv 6fe473217eda2e9b87c5291d28ff6e14339107b1b88346a6f72e62473713ed4a; tables/Table4_HG008_sensitivity_all_region.tsv f838cab270034cc351c5fa0fd7442b7ca6318beca598640b988eaa3952123ef8 (each equals the archive's own SHA256SUMS entry). Inspected artifact Extraction artifact SHA-256: `629921b1198f4a5290d191857d13f98e54a1d132d722caf0ac83805f7e24a40d` Extraction artifact

How to read the evidence

Sources and history

Release 2026-10-22-1b7cf027e4f6 · Record review: source checked

2 source records and release history

Prospectively frozen external benchmarking reveals benchmark-context dependence in somatic variant calling · Original source · Research Square preprint rs-11210923 version 1, posted 2026-10-05; manuscript PDF (16 pages)
AmbioSpire P003/T001 publication source data and reproducibility package v1.0.0 (Zenodo) · Original source · Zenodo record 22998005, version 1.0.0, published 2026-09-27; file P003-T001-publication-data-v1.0.0-zenodo-final.zip (1,934,562 bytes, md5 a1fdab6108197602f07f72db77d9f996)

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: somatic-hg008-20261011-result-majority-all-region-overall-f1-score

metric: f1-score
metric direction: higher
unit: fraction
printed value: 0.793961
source locator: Zenodo tables/Table4_HG008_sensitivity_all_region.tsv, row 'Majority', column 'F1'
review: method: source-hash-verification; deterministic-table-parse; independent-cell-check; method note: Independent re-retrieval and hash check; reviewer's own parser compared every table cell; stratified hand check against the rendered table or quoted sentence.; reviewer: claude; reviewer note: Claude research agent reviewer-onc-1, independent of the extractor uc-oncology; no human review claimed; date: 2026-10-11; artifact sha256: 629921b1198f4a5290d191857d13f98e54a1d132d722caf0ac83805f7e24a40d; retrieval url: https://zenodo.org/api/records/22998005/files/P003-T001-publication-data-v1.0.0-zenodo-final.zip/content; note: Extraction: Read tables/Table4_HG008_sensitivity_all_region.tsv from the pinned Zenodo zip with extract/extract_hg008.py; member SHA-256 checked against the archive's SHA256SUMS; method and class row labels asserted. Zenodo tables/Table4_HG008_sensitivity_all_region.tsv, row 'Majority', column 'F1' Independent review 2026-10-11: value, locator, metric, unit, direction and linked configuration and protocol checked; corrections from the review applied by the extractor as listed in review.md (docs/reviews/use-cases/tumour-dna-somatic-variant-detection-2026-10-11.md).
numeric value: 0.793961
metric qualifier: SNVs and indels, HG008 all-benchmark region
uncertainty: type: confidence_interval; lower: 0.7875; upper: 0.8003; level: 0.95; method: bootstrap_percentile; resamples: 10000; printed: 0.7875–0.8003; source column: Manuscript Results, 'Sensitivity analyses show benchmark-context dependence without establishing population generalization', paragraph 2 (Supplementary Table 2); note: Genomic block bootstrap over 5-Mb blocks, linear percentile interval; prespecified, but executed after the primary external evaluation; spatial variation within the benchmark region, not between tumours
printed value basis: Value as stored in the author's Zenodo table (six decimals); the manuscript prints some of these rounded to four decimals

Related records

evaluation: Majority on HG008 all-benchmark region (sensitivity analysis), SNVs and indels combined (Hasan 2026)

Suggest a correction