rewirebio.iobenchmarks

Use caseDNA and genomesResearch and clinical research

Select a tumour DNA somatic variant-calling workflow

Which calling or rescoring workflow reliably detects tumour SNVs and small indels under the available sequencing and normal-sample regime?

What the evidence shows

Eight studies are linked, and their values cannot be pooled: truth sets, sequencing, caller versions and scoring scripts differ.

Values are rounded to two decimals, except where the source prints percentages. In the Lancet developers' own virtual-tumour test, Lancet and Strelka2 both reached SNV F1 0.85, with indel F1 0.81 and 0.77. Wang et al. 2020 ran eight callers, NeuSomatic and their own majority-vote ensembles on eight whole-genome tumour-normal pairs, with older caller versions (for example MuTect2 from GATK 3.7).

An ensemble had the highest SNV F1 on seven of the eight pairs; on DREAM set 4, NeuSomatic, whose model was trained on DREAM set 3, was highest at 0.86. The best single configuration reached SNV F1 0.92 to 0.96 on the simulated DREAM sets 1 to 3 but 0.70 to 0.81 on the CLL, AML and MB tumours, where most callers had precision below 0.6 against curated truth sets that the authors say may be incomplete, so precision there is a lower bound. Indel F1 on the CLL and MB tumours and the COLO829 cell line (AML has no indel truth set) was at most 0.72 for every configuration.

Guille et al. 2025, who did not develop the callers, scored them on a WES replicate of the SEQC2 HCC1395 cell line (1,160 truth SNVs, 50 truth indels); another replicate with the same truth set was used to choose their ensembles. Leaving out DeepSomatic and NeuSomatic, whose pre-trained models were built with HCC1395 data, the best single caller reached SNV F1 0.85 (Mutect) and indel F1 0.66 (VarNet), and the authors' ensembles reached 0.88 for SNVs and 0.75 and 0.74 for indels. Strelka reached SNV F1 0.64 there, with precision 0.51, against 0.85 for Strelka2 in the Lancet test, which shows how much results depend on the study.

Sahraeian et al. 2022, the NeuSomatic developers, scored callers on SEQC2 HCC1395 whole-genome data in the half of the genome held out from their model training: 21 replicate pairs from six centres, 47 tumour purity and coverage mixtures, and six library preparations. Their NeuSomatic and NeuSomatic-S models trained on SEQC2 data (SEQC-WGS-Spike, SEQC-WGS-GT-50 and SEQC-WGS-GT50-SpikeWGS10) and Octopus-RF, whose random forest is not identified, are left out because they were built from the same cell line or may have been. Among the rest, mean F1 over the 21 replicates was 92.3% for DRAGEN, 91.8% for NeuSomatic with its DREAM3 model and 89.0% for Strelka2 for SNVs, and 81.7%, 79.3% and 72.8% for indels.

Over the purity and coverage mixtures (10x to 300x, 5% to 100% tumour), mean SNV F1 was at most 67.7% (Octopus with hard filters). With 5% tumour in an 80x normal, SNV F1 for a pure tumour fell from 88.6% to 43.8% for MuTect2 and from 93.3% to 47.8% for Lancet, but only from 92.4% to 90.9% for Strelka2. Hasan 2026, a single-author preprint run under the author's own company by someone who developed none of the callers, scored Mutect2, Strelka2, DeepSomatic and three ensembles defined by the author on the GIAB HG008 tumour-normal pair against a draft truth set of clonal and truncal variants; the original scoring code was not retained.

In the primary region (14,014 truth variants, excluding sites with germline interference), F1 was 0.91 for a two-of-three majority vote, 0.90 for Mutect2 and DeepSomatic and 0.78 for Strelka2; for indels Mutect2 reached 0.82 and DeepSomatic and Strelka2 0.69. Over the whole draft benchmark, F1 was 0.70 to 0.79 for all six. Two developer papers tested tumour-only calling, without the matched normal this use case assumes, so they are linked as proxy evidence.

On Illumina COLO829 at 50x, ClairS-TO's developers report SNV F1 0.77 for ClairS-TO against 0.62 for DeepSomatic, 0.58 for Mutect2, 0.40 for Octopus and 0.22 for Pisces, with indel F1 at most 0.29; each F1 is at the threshold that gave that caller its best F1 on the test data, and truth variants with fewer than three supporting reads or VAF below 0.05 in the tested data were not scored. The VarNet-T developers report SNV AUPRC 0.77 for VarNet-T on SEQC2 and 0.66 on COLO829, against 0.54 and 0.27 for ClairS-TO and 0.08 and 0.07 for Mutect2; on three low-burden tumours every tumour-only caller had AUPRC at most 0.11. The two papers used different COLO829 sequencing data and truth sets: the same ClairS-TO model (Illumina SSRS, v0.4.0) had COLO829 SNV AUPRC 0.65 in its developers' paper and 0.27 in the VarNet-T paper.

On a synthetic WES pair, where inserted SNVs make precision near 1 by construction, Mutect2 recalled 0.63 of inserted SNVs and Strelka2 0.46 with default nf-core/sarek settings. Only the HG008 preprint reports interval estimates; no linked study covers FFPE tissue, ctDNA or targeted panels, and none establishes clinical performance.

You bring
  • Tumour and matched-normal aligned sequencing reads
  • The intended variant-fraction, coverage and genomic-context strata
You want
A sourced set of exact evaluated caller precision/recall/F1 values and remaining validation needs; no patient-level variant classification or clinical recommendation.
Evidence in this release
11 comparisons shown · 27 tools · 82 configurations
Kept up to date
Evidence last checked 11 October 2026 · re-checked monthly, next due 11 November 2026
Executive Takeaway

For "Select a tumour DNA somatic variant-calling workflow", on SEQC2 HCC1395 WES validation sample (Guille et al. 2025), SNVs (F1), Lofreq-Muse-Mutect2-SomaticSniper-Strelka-Lancet scored 88.1%, reported by A benchmarking study of individual somatic variant callers and voting-based ensembles for whole-exome sequencing.

Best reported valueLofreq-Muse-Mutect2-SomaticSniper-Strelka-LancetA benchmarking study of individual somatic variant callers…
F188.1%SEQC2 HCC1395 WES validation sample (Guille et al. 2025), SNVs
Baseline on this protocolNone scoredNo margin is computed
Ranked tools1614 from independent studies

16 tools have a value on this protocol; 14 of them come from studies whose authors did not develop the tool. No baseline method is scored on this protocol.

Ranking

Ranked on SEQC2 HCC1395 WES validation sample (Guille et al. 2025), SNVs (F1, higher is better). Only results on this protocol are ranked; other tools are listed below the table.

15 tools in 16 rows, from 2 sources. 14 of 16 results (88%) come from an independent study; 2 are reported by the tool's developers. The top-ranked result is reported by the tool's developers.

Rank▲ModelF1 (higher is better)EvidenceCoverageReported byLicense
#188.1%Source-checked Developer-reported4/11A benchmarking study of individual somatic…Not recorded
#287.5%Source-checked Developer-reported4/11A benchmarking study of individual somatic…Not recorded
#385.3%Source-checked Independent study2/11A benchmarking study of individual somatic…Not recorded
#484.7%Source-checked Independent study3/11A benchmarking study of individual somatic…Not recorded
#584.0%Source-checked Independent study1/11A benchmarking study of individual somatic…Not recorded
#684.0%Source-checked Independent study5/11A benchmarking study of individual somatic…Not recorded
#783.5%Source-checked Independent study1/11A benchmarking study of individual somatic…Not recorded
#8
Mutect2[-] Baseline
83.3%Source-checked Independent study10/11A benchmarking study of individual somatic…Not recorded
#977.2%Source-checked Independent study1/11A benchmarking study of individual somatic…Not recorded
#1075.4%Source-checked Independent study6/11A benchmarking study of individual somatic…Not recorded
#1173.3%Source-checked Independent study3/11A benchmarking study of individual somatic…Not recorded
#1271.7%Source-checked Independent study5/11A benchmarking study of individual somatic…Not recorded
#1365.9%Source-checked Independent study4/11A benchmarking study of individual somatic…Not recorded
#14
Strelka[-] Baseline
63.9%Source-checked Independent study8/11A benchmarking study of individual somatic…Not recorded
#1559.6%Source-checked Independent study1/11A benchmarking study of individual somatic…Not recorded
#1633.1%Source-checked Independent study1/11A benchmarking study of individual somatic…Not recorded
13 tools are evaluated elsewhere in this use case but not on this protocol, so not ranked

Baseline

Mutect2 and Strelka2. Mutect2 and Strelka2 are widely used tumour-normal callers, scored in most linked somatic benchmarks. Both call variants with statistical models of read evidence rather than trained classifiers.

Machine-learning methods are compared with the baseline on the same data in 9 tables: SEQC2 HCC1395 WES validation sample (Guille et al. 2025), WGS somatic SNVs by dataset (Wang et al. 2020), WGS somatic indels by dataset (Wang et al. 2020), Tumour-only callers on cell lines (Chen et al. 2025, ClairS-TO developers), Mutect2, Strelka2, DeepSomatic and ensembles on HG008 (Hasan 2026), Library preparation and DNA input (SEQC2 HCC1395) (Sahraeian et al. 2022), Tumour purity, coverage and normal contamination (SEQC2 HCC1395 titration) (Sahraeian et al. 2022), Sequencing centre and platform (SEQC2 HCC1395 WGS replicates) (Sahraeian et al. 2022), Tumour-only callers on Illumina WGS (Krishnamachari et al. 2026, VarNet-T developers). Baseline rows are marked in each table.

Tools compared

Comparisons

Unified model comparison for this use case. Compare tools against protocol baselines in a single view.

SEQC2 HCC1395 WES validation sample (Guille et al. 2025)

Direct evidenceDeveloper-reported: Reported by the developers of the evaluated tool.Independent study: Reported in a study whose authors did not develop the evaluated tool.

F1 (higher is better), with the other metrics below each value. Showing the top 10 of 17 rows. The best value in each column is highlighted.

ToolSNVsIndels
88.1% (best in column)Not reported
Mutectconventional
85.3%Not reported
Lofreqconventional
84.7%32.65%
VarNetmachine learning
84.0%65.88%
Museconventional
84.0%Not reported
Virmidconventional
83.5%Not reported
Mutect2Baselineconventional
83.3%65.04%
Seuratconventional
77.2%15.30%
Vardictconventional
75.4%20.43%
Varscan2conventional
73.3%41.43%
SomaticSniperconventional
71.7%Not reported
Lancetconventional
65.9%46.25%
StrelkaBaselineconventional
63.9%53.66%
FreeBayesconventional
59.6%17.82%
Shimmerconventional
33.1%Not reported
Pindelconventional
Not reported46.15%
Scalpelconventional
Not reported5.30%
Show all 17 rowsShow the top 10 rows only

R recall, P precision.

Caveats and method

What each column measures:

  • SNVs: TP, FP, recall, false-positive rate, precision and F1 for somatic SNVs on a held-out SEQC2 HCC1395 WES sample for 16 individual callers and 2 retained voting ensembles
  • Indels: TP, FP, recall, false-positive rate, precision and F1 for somatic indels on a held-out SEQC2 HCC1395 WES sample for 13 individual callers and 2 retained voting ensembles
  • Single WES sample of one cell line; no replicate or interval estimates.
  • Truth limited to SEQC2 high-confidence regions within the exome (1,160 SNVs).
  • Not fully independent of development: the ensembles were selected on four datasets that include another WES replicate of the same HCC1395 pair with the same truth set (Table 2 row 'SEQC2').
  • Post-alignment procedure for Table S7 is not stated (Tables S1-S2 compare four procedures on the development datasets).
  • DeepSomatic and NeuSomatic pre-trained models were built with SEQC2 HCC1395 data (Discussion paragraph 5; supplementary methods checkpoint name), so their rows are not held out.
  • The authors' 'best individual tool' comparison in Results 'Validation' covers the classic callers only (gains of 2.7 and 10.2 points over Mutect and Mutect2); it excludes the deep-learning callers, as the ensemble search did (Methods 'Ensemble approach').
  • Small truth set: 50 indels in SEQC2 high-confidence regions, so one variant moves recall by 0.02.

Protocols: SEQC2 HCC1395 WES validation sample, somatic SNVs (Guille et al. 2025 Table S7), SEQC2 HCC1395 WES validation sample, somatic indels (Guille et al. 2025 Table S7)

Primary benchmark evaluation shown (SEQC2 HCC1395 WES validation sample (Guille et al. 2025)). 10 additional study protocols (Lancet paper virtual-tumor SNV/indel benchmark, WGS somatic SNVs by dataset (Wang et al. 2020), WGS somatic indels by dataset (Wang et al. 2020) and others) are cited in individual tool drawer rows and under Question, sources and review.

Not covered yet

  • Lancet paper: individual-condition confidence intervals are unreported in the source tables.
  • Lancet paper: exact caller release versions are not extracted from the primary article.
  • No foundation-model component is evaluated in this bounded intake.
  • Tumour-only calling is covered only by proxy evidence from two developer papers (ClairS-TO, VarNet-T) on cell lines and research tumours; no linked comparison covers FFPE tissue, ctDNA or targeted panels.
  • The only linked HG008 small-variant caller comparison is a single-author preprint scored on the draft HG008-T v0.3 benchmark.
  • The SEQC2 consortium's own whole-genome caller comparison (Xiao et al. 2021) was not retrieved (not open access through Europe PMC); Sahraeian et al. 2022 Tables S5 to S10 (WES, AmpliSeq, FFPE, sample-specific models, runtime) are not linked.
  • Guille et al. 2025 Tables S1-S2 (four development datasets, including PERMED-01 clinical breast cancer exomes, under four post-alignment procedures) and Wang et al. 2020 Tables S3-S8 (deep targeted and exome dilution series) are not transcribed in this bounded intake.
  • Only the HG008 preprint (Hasan 2026) reports interval estimates, for F1 only (genomic block bootstrap within one sample); no other linked study reports interval estimates for per-caller precision, recall or F1.
  • Caller versions differ between studies (for example MuTect2 from GATK v3.7-0 in Wang et al. 2020 and Mutect2 4.2.2.0 in Guille et al. 2025); no study evaluates current releases of every caller.
  • The SEQC2 whole-genome comparison is a developer study on one cell line, and the developers' SEQC2-trained NeuSomatic models and Octopus-RF are excluded for training on that cell line; no linked comparison tests those models on an independent tumour.
  • Both linked tumour-only comparisons are developer papers (ClairS-TO, VarNet-T); no independent tumour-only comparison with printed per-caller values was found in the 2025-10 to 2026-10 window.
  • Tumour-only precision, recall and F1 in Chen et al. 2025 are at a threshold chosen on the test data; Krishnamachari et al. 2026 is extracted as AUPRC only.
  • The two tumour-only papers used different COLO829 sequencing data and truth sets (NYGC in Chen et al. 2025, TGen in Krishnamachari et al. 2026), so their COLO829 values are not comparable.
  • Long-read tumour-normal calling (ClairS, Nature Methods 2026) and the peer-reviewed DeepSomatic paper (Nature Biotechnology 2025, not open access) are not linked.
  • The amplicon-panel tumour-only comparison found in this pass (Bharne and Gaston 2026 preprint, HD789 reference standard) prints detection per variant but no per-caller performance metric, so panels remain uncovered.

Contribute evidence or propose a correction

How to cite

Select a tumour DNA somatic variant-calling workflow. Rewire biological model benchmark database, release 2026-11-21-adc15377fab2. https://benchmarks.rewirebio.io/use-cases/tumour-dna-somatic-variant-detection/

Release 2026-11-21-adc15377fab2. Rankings and values change between releases, so keep the release id in the citation. Cite the original publication as well for any value you reuse: each result links its source.

The compilation and annotations made by Rewire in this release are offered under CC BY 4.0; values and text taken from cited sources stay under the terms of those sources and are not covered by this licence. Data licence

Question, sources and review

Who this is for, setting and exclusions

Inspect the per-dataset precision, recall and F1 of the callers and ensembles in the linked comparisons, matched to the intended sequencing type, coverage and normal-sample regime, before selecting a caller and designing a validation protocol.

Who this is for
  • Clinical researchers scoping the available diagnostic-genomics evidence
  • Computational researchers comparing exact evaluated configurations
Biological setting
Published tumour-normal comparisons: a real-read virtual-tumour spike-in (Lancet paper, 80x/40x WGS), four simulated DREAM WGS pairs and four real tumour or cell-line WGS pairs with curated truth sets (Wang et al. 2020), and a validation WES replicate of the SEQC2 HCC1395 cell line (Guille et al. 2025), each measuring per-caller SNV and indel precision, recall and F1.
Clinical research scope
Clinical applicability is not established. The linked studies score callers on synthetic tumours, cell lines and research tumour samples against research truth sets; they do not validate any caller for clinical use, FFPE specimens, gene panels or ctDNA, or for copy-number or structural variants.

Outside this use case

  • FFPE samples, targeted panels, ctDNA, tumour-only calling, copy-number and structural variation
  • Lancet paper: its Strelka, MuTect, MuTect2 and LoFreq rows are not ingested; the unselected LoFreq and MuTect2 rows are internally inconsistent in the source.
  • Callers evaluated only with a pre-trained model built from the test cell line or simulator (excluded from the affected judgements).
Evidence collection plan
Evidence collection planCollecting evidence

Already on record: 11 comparisons shown · 27 tools, in the evaluated evidence above. The status above describes only the specific comparison in this plan, which remains open; it does not mean no evidence has been collected. The plan defines a comparison to investigate; it does not establish model performance or suitability.

Comparison question

Which calling or rescoring workflow reliably detects tumour SNVs and small indels under the available sequencing and normal-sample regime?

Baselines, outcomes and validation requirements

Baselines to include

  • The conventional/author-introduced workflow measured in the linked primary source(s).

Outcomes to measure

  • The declared endpoint in this use case's active mapping(s); see evidence_gaps for what remains open.

Validation requirements

  • Independent held-out population matched to the intended clinical setting.
  • Qualified human scientific review before any clinical-validation claim.

Next collection task

A dedicated rewire-benchmarks protocol/run task with a real clinical tumour cohort, pinned truth set, purity/platform/assembly strata and an explicit foundation-model eligibility check.

Sources, review and downloads

Automated source review · 2026-10-07 · Claude Sonnet AMP-integration worker, bounded transcription of Codex-checked primary values; independently reviewed by Codex (workbench/amp-supervision/primary-review.md, integration-review-corrections.md)

Bounded primary-source transcription, independently Codex-checked. No new model execution, independent experimental reproduction, qualified human scientific review or clinical validation.

Release 2026-11-21-adc15377fab2 · question use-case-tumour-dna-somatic-variant-detection

Use-case input digest bfbb7ab9303becad1e3cb395568c26f41e72ddb67cd1a89a06d9eaa0f76bd95f

Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)