Prospectively frozen external benchmarking reveals benchmark-context dependence in somatic variant calling

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 6 configurations
• 4 protocols
• 1 dataset
• 24 evaluations
• 144 results
• 5 evidence claims
• 1 use case

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• Evidence summary: tumour DNA somatic SNV and indel calling: Summary Eight studies are linked, and their values cannot be pooled: truth sets, sequencing, caller versions and scoring scripts differ. Values are rounded to two decimals, except where the source prints percentages. In the Lancet developers' own virtual-tumour test, Lancet and Strelka2 both reached SNV F1 0.85, with indel F1 0.81 and 0.77. Wang et al. 2020 ran eight callers, NeuSomatic and their own majority-vote ensembles on eight whole-genome tumour-normal pairs, with older caller versions (for example MuTect2 from GATK 3.7). An ensemble had the highest SNV F1 on seven of the eight pairs; on DREAM set 4, NeuSomatic, whose model was trained on DREAM set 3, was highest at 0.86. The best single configuration reached SNV F1 0.92 to 0.96 on the simulated DREAM sets 1 to 3 but 0.70 to 0.81 on the CLL, AML and MB tumours, where most callers had precision below 0.6 against curated truth sets that the authors say may be incomplete, so precision there is a lower bound. Indel F1 on the CLL and MB tumours and the COLO829 cell line (AML has no indel truth set) was at most 0.72 for every configuration. Guille et al. 2025, who did not develop the callers, scored them on a WES replicate of the SEQC2 HCC1395 cell line (1,160 truth SNVs, 50 truth indels); another replicate with the same truth set was used to choose their ensembles. Leaving out DeepSomatic and NeuSomatic, whose pre-trained models were built with HCC1395 data, the best single caller reached SNV F1 0.85 (Mutect) and indel F1 0.66 (VarNet), and the authors' ensembles reached 0.88 for SNVs and 0.75 and 0.74 for indels. Strelka reached SNV F1 0.64 there, with precision 0.51, against 0.85 for Strelka2 in the Lancet test, which shows how much results depend on the study. Sahraeian et al. 2022, the NeuSomatic developers, scored callers on SEQC2 HCC1395 whole-genome data in the half of the genome held out from their model training: 21 replicate pairs from six centres, 47 tumour purity and coverage mixtures, and six library preparations. Their NeuSomatic and NeuSomatic-S models trained on SEQC2 data (SEQC-WGS-Spike, SEQC-WGS-GT-50 and SEQC-WGS-GT50-SpikeWGS10) and Octopus-RF, whose random forest is not identified, are left out because they were built from the same cell line or may have been. Among the rest, mean F1 over the 21 replicates was 92.3% for DRAGEN, 91.8% for NeuSomatic with its DREAM3 model and 89.0% for Strelka2 for SNVs, and 81.7%, 79.3% and 72.8% for indels. Over the purity and coverage mixtures (10x to 300x, 5% to 100% tumour), mean SNV F1 was at most 67.7% (Octopus with hard filters). With 5% tumour in an 80x normal, SNV F1 for a pure tumour fell from 88.6% to 43.8% for MuTect2 and from 93.3% to 47.8% for Lancet, but only from 92.4% to 90.9% for Strelka2. Hasan 2026, a single-author preprint run under the author's own company by someone who developed none of the callers, scored Mutect2, Strelka2, DeepSomatic and three ensembles defined by the author on the GIAB HG008 tumour-normal pair against a draft truth set of clonal and truncal variants; the original scoring code was not retained. In the primary region (14,014 truth variants, excluding sites with germline interference), F1 was 0.91 for a two-of-three majority vote, 0.90 for Mutect2 and DeepSomatic and 0.78 for Strelka2; for indels Mutect2 reached 0.82 and DeepSomatic and Strelka2 0.69. Over the whole draft benchmark, F1 was 0.70 to 0.79 for all six. Two developer papers tested tumour-only calling, without the matched normal this use case assumes, so they are linked as proxy evidence. On Illumina COLO829 at 50x, ClairS-TO's developers report SNV F1 0.77 for ClairS-TO against 0.62 for DeepSomatic, 0.58 for Mutect2, 0.40 for Octopus and 0.22 for Pisces, with indel F1 at most 0.29; each F1 is at the threshold that gave that caller its best F1 on the test data, and truth variants with fewer than three supporting reads or VAF below 0.05 in the tested data were not scored. The VarNet-T developers report SNV AUPRC 0.77 for VarNet-T on SEQC2 and 0.66 on COLO829, against 0.54 and 0.27 for ClairS-TO and 0.08 and 0.07 for Mutect2; on three low-burden tumours every tumour-only caller had AUPRC at most 0.11. The two papers used different COLO829 sequencing data and truth sets: the same ClairS-TO model (Illumina SSRS, v0.4.0) had COLO829 SNV AUPRC 0.65 in its developers' paper and 0.27 in the VarNet-T paper. On a synthetic WES pair, where inserted SNVs make precision near 1 by construction, Mutect2 recalled 0.63 of inserted SNVs and Strelka2 0.46 with default nf-core/sarek settings. Only the HG008 preprint reports interval estimates; no linked study covers FFPE tissue, ctDNA or targeted panels, and none establishes clinical performance. (amp-oncology-rna-20261007-source-pmc6123722: Lancet virtual-tumour results; somatic-20261009-source-wang2020-tables: Tables S1-S2; somatic-20261009-source-guille2025-table-s7: Table S7; somatic-20261009-source-wang2020 and somatic-20261009-source-guille2025: Methods, Results and Discussion; somatic-neusomatic-20261010-source-sahraeian2022-additional-file-2: Tables S1-S4; somatic-neusomatic-20261010-source-sahraeian2022: Methods; somatic-clairs-to-20261011-source-chen2025: see claims.csv of its batch; somatic-clairs-to-20261011-source-chen2025-source-data: see claims.csv of its batch; somatic-clairs-to-20261011-source-chen2025-supplementary: see claims.csv of its batch; somatic-varnet-t-20261011-source-krishnamachari2026: see claims.csv of its batch; somatic-varnet-t-20261011-source-krishnamachari2026-source-data: see claims.csv of its batch; somatic-varnet-t-20261011-source-krishnamachari2026-supplementary: see claims.csv of its batch; somatic-hg008-20261011-source-hasan2026: see claims.csv of its batch; somatic-hg008-20261011-source-hasan2026-zenodo: see claims.csv of its batch; somatic-wes-synthetic-20261011-source-lopezcade2025: see claims.csv of its batch)
• Relevance of HG008 all-benchmark region (sensitivity analysis), SNVs and indels combined (Hasan 2026) to "Select a tumour DNA somatic variant-calling workflow": Assessed by HG008 all-benchmark region (sensitivity analysis), SNVs and indels combined, Mutect2, Strelka2, DeepSomatic and ensembles (Hasan 2026 Table 4) (somatic-hg008-20261011-source-hasan2026: Results 'Caller and deterministic-consensus performance differ in the HG008 external benchmark'; Online Methods; Declarations; somatic-hg008-20261011-source-hasan2026-zenodo: tables/Table1_HG008_primary_metrics.tsv, Table2_HG008_variant_class_metrics.tsv, Table4_HG008_sensitivity_all_region.tsv)
• Relevance of HG008 primary region (nogermlineinterference), indels (Hasan 2026) to "Select a tumour DNA somatic variant-calling workflow": Assessed by HG008 primary region (nogermlineinterference), indels, Mutect2, Strelka2, DeepSomatic and ensembles (Hasan 2026 Table 2, Class 'INDEL') (somatic-hg008-20261011-source-hasan2026: Results 'Caller and deterministic-consensus performance differ in the HG008 external benchmark'; Online Methods; Declarations; somatic-hg008-20261011-source-hasan2026-zenodo: tables/Table1_HG008_primary_metrics.tsv, Table2_HG008_variant_class_metrics.tsv, Table4_HG008_sensitivity_all_region.tsv)
• Relevance of HG008 primary region (nogermlineinterference), SNVs (Hasan 2026) to "Select a tumour DNA somatic variant-calling workflow": Assessed by HG008 primary region (nogermlineinterference), SNVs, Mutect2, Strelka2, DeepSomatic and ensembles (Hasan 2026 Table 2, Class 'SNV') (somatic-hg008-20261011-source-hasan2026: Results 'Caller and deterministic-consensus performance differ in the HG008 external benchmark'; Online Methods; Declarations; somatic-hg008-20261011-source-hasan2026-zenodo: tables/Table1_HG008_primary_metrics.tsv, Table2_HG008_variant_class_metrics.tsv, Table4_HG008_sensitivity_all_region.tsv)
• Relevance of HG008 primary region (nogermlineinterference), SNVs and indels combined (Hasan 2026) to "Select a tumour DNA somatic variant-calling workflow": Assessed by HG008 primary region (nogermlineinterference), SNVs and indels combined, Mutect2, Strelka2, DeepSomatic and ensembles (Hasan 2026 Table 1) (somatic-hg008-20261011-source-hasan2026: Results 'Caller and deterministic-consensus performance differ in the HG008 external benchmark'; Online Methods; Declarations; somatic-hg008-20261011-source-hasan2026-zenodo: tables/Table1_HG008_primary_metrics.tsv, Table2_HG008_variant_class_metrics.tsv, Table4_HG008_sensitivity_all_region.tsv)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.