rewirebio.iobenchmarks

Use caseDNA and genomesResearch and clinical research

Select a DNA pathogen-identification workflow for diagnostic testing

Which DNA sequencing classification workflow detects clinically relevant pathogens and handles contamination or organisms absent from the reference?

What the evidence shows

Most workflow comparisons in the reviewed evidence use mock communities, constructed metagenomes or simulated sequences.

Two compare workflows on patient specimens, and in both the reference standard draws partly on sequencing: in Hu et al. 2026 only the culture comparison is independent of the sequencing results, and in Huang et al. 2026 the reference was built mainly from BALF mNGS and the conventional tests that are themselves scored. Both are author-reported and have co-authors from sequencing companies (Genskey Medical Technology for Hu; Geneplus-Beijing, which supplies the targeted assay's probes, for Huang). For the Karius plasma cell-free DNA assay, the developers report 93.7% (59 of 63) agreement with initial blood culture in a sepsis-alert cohort, and blood culture does not detect every infection.

A prospective study in children and young adults with high-risk leukaemia, with Karius taking part in sample processing and analysis, found every blood-culture organism in 81.3% (65 of 80) of evaluable bloodstream infection episodes on the day of or after onset and in 51.9% (42 of 81) during the three days before; 82.7% (134 of 162) of control samples from patients without fever or infection had no bacterium or yeast detected. In 62 bronchoalveolar lavage specimens (Hu et al. 2026), a short-read MGI workflow and a host-depleted nanopore workflow both had 95.2% positive agreement with culture but 44.6% and 46.4% negative agreement, because they found organisms culture did not; against a composite reference that adds organisms found by both workflows, positive agreement was 93.3% and 90.7%. In 122 intensive-care patients with pneumonia-associated sepsis (Huang et al. 2026), a targeted probe-capture assay on plasma had 63.46% sensitivity against the reference diagnosis, against 35.58% for untargeted blood mNGS, 49.04% for conventional tests and 9.62% for blood culture, with specificity of 44.44% and 61.11% for the two blood sequencing assays.

In mock communities (Portik et al. 2022), Kraken2 and Bracken called 44 to 204 absent species at a 0.001% read threshold (precision 0.05 to 0.21), while long-read MEGAN-LR and BugSeq-V2 called 0 to 2; the first author is a PacBio employee. In seven pathogen-free libraries scored against one shared virus database by the EsViritu developers (preprint), the median number of false virus species was 0 for EsViritu, Centrifuger and Sylph, 6 for Kraken2 and 18 for Metabuli. For M. tuberculosis reads in constructed metagenomes (Hall and Coin 2024), kraken with its standard database classified 0.07 to 0.74 of the M. tuberculosis reads as M. tuberculosis, against 0.97 to 0.995 with a Mycobacterium database.

Two papers by the developers of trained classifiers compare them with the baseline tools, on simulated or isolate-derived data rather than clinical specimens. On simulated 1,500-nt genome fragments (Mock et al. 2022, BERTax), Kraken2 and minimap2 assigned a superkingdom to 76.11% and 75.59% of fragments, with 99.82% and 99.95% accuracy on the fragments they assigned, when relatives were in the reference, and to 21.56% and 19.88% with 90.71% and 94.42% accuracy when each fragment's genus was absent from it; BERTax assigned every fragment, with 94.78% and 88.95% accuracy in the two settings. On in silico mixtures of nanopore isolate reads from species absent from the reference (Peres da Silva et al. 2024, MetageNN), MetageNN assigned a genus to 93.26% of reads on average across 30 mixtures, against 64.90% for Kraken2 and 53.06% for MEGAN-LR (on minimap2 alignments); MMseqs2, a reference-based tool not in the baseline, assigned 85.49%.

These shares count reads assigned, not reads assigned correctly. Each study used its own reference database and reporting rules, and none of these results is clinical validation.

You bring
  • Plasma microbial cell-free DNA sequencing reads
  • The reference standard available for agreement/sensitivity comparison (e.g. blood culture, composite adjudication)
You want
A sourced positive-percent-agreement figure against one reference standard, with the reference standard's own detection limits explicit; no general sensitivity claim against all true infections.
Evidence in this release
9 comparisons shown · 2 held for review · 27 tools · 66 configurations
Kept up to date
Evidence last checked 11 October 2026 · re-checked monthly, next due 11 November 2026
Executive Takeaway

For "Select a DNA pathogen-identification workflow for diagnostic testing", on Species detection in mock communities (Portik et al. 2022), PacBio HiFi, ATCC MSA-1003 (F1, a proxy protocol), Sourmash-k51 scored 0.93, reported by Evaluation of taxonomic classification and profiling methods for long-read shotgun metagenomic sequencing datasets.

Best reported valueSourmash-k51Evaluation of taxonomic classification and profiling…
F10.93Species detection in mock communities (Portik et al. 2022), PacBio HiFi, ATCC MSA-1003
Baseline on this protocolNone scoredNo margin is computed
Ranked tools1412 from independent studies

14 tools have a value on this protocol; 12 of them come from studies whose authors did not develop the tool. No baseline method is scored on this protocol. Reviewers judged this protocol a proxy for the use case.

Ranking

Ranked on Species detection in mock communities (Portik et al. 2022), PacBio HiFi, ATCC MSA-1003 (F1, higher is better). Proxy protocol Reviewers judged this protocol a proxy for the use case, not a direct test of it. Only results on this protocol are ranked; other tools are listed below the table.

10 tools in 14 rows, from 1 source. 12 of 14 results (86%) come from an independent study; 2 are reported by the tool's developers. The top-ranked result is reported by the tool's developers.

Rank▲ModelF1 (higher is better)EvidenceCoverageReported byLicense
#10.93Source-checked Developer-reported2/9Evaluation of taxonomic classification and…Not recorded
#20.89Source-checked Independent study1/9Evaluation of taxonomic classification and…Not recorded
#20.89Source-checked Developer-reported2/9Evaluation of taxonomic classification and…Not recorded
#40.88Source-checked Independent study3/9Evaluation of taxonomic classification and…MIT
#5
MEGAN-LR-Prot[-] Baseline
0.85Source-checked Independent study2/9Evaluation of taxonomic classification and…Not recorded
#6
MEGAN-LR-Nuc-HiFi[-] Baseline
0.82Source-checked Independent study2/9Evaluation of taxonomic classification and…Not recorded
#6
MEGAN-LR-Nuc-ONT[-] Baseline
0.82Source-checked Independent study2/9Evaluation of taxonomic classification and…Not recorded
#80.79Source-checked Independent study1/9Evaluation of taxonomic classification and…Not recorded
#90.78Source-checked Independent study2/9Evaluation of taxonomic classification and…Not recorded
#100.73Source-checked Independent study1/9Evaluation of taxonomic classification and…Not recorded
#110.71Source-checked Independent study1/9Evaluation of taxonomic classification and…Not recorded
#120.37Source-checked Independent study1/9Evaluation of taxonomic classification and…MIT
#130.29Source-checked Independent study5/9Evaluation of taxonomic classification and…MIT
#140.26Source-checked Independent study1/9Evaluation of taxonomic classification and…Not recorded
17 tools are evaluated elsewhere in this use case but not on this protocol, so not ranked

Baseline

Kraken2 and minimap2 with MEGAN-LR. Kraken2 matches read k-mers against a reference database, and minimap2 and MEGAN-LR assign reads by alignment to reference sequences; none is a trained model. They are the conventional classifiers scored across the linked benchmarks, and in Portik et al. 2022 alignment-based long-read methods such as MEGAN-LR had high precision and recall without filtering.

Machine-learning methods are compared with the baseline on the same data in 2 tables: BERTax genome-fragment classification against database tools (Mock et al. 2022), MetageNN against alignment and k-mer classifiers on ONT reads (Peres da Silva et al. 2024). Baseline rows are marked in each table.

Tools compared

Comparisons

Unified model comparison for this use case. Compare tools against protocol baselines in a single view.

Species detection in mock communities (Portik et al. 2022)

Proxy evidenceDeveloper-reported: Reported by the developers of the evaluated tool.Independent study: Reported in a study whose authors did not develop the evaluated tool.

F1 (higher is better), with the other metrics below each value. The best value in each column is highlighted.

ToolPacBio HiFi, ATCC MSA-1003Illumina, ATCC MSA-1003PacBio HiFi, Zymo D6331ONT R10.3, Zymo D6300ONT Q20, Zymo D6300Illumina, Zymo D6300
Sourmash-k512 sweepsconventional
0.93 (best in column)0.89 (best in column)0.900.870.830.86
BugSeq-V2conventional
0.89Not reported0.890.950.95Not reported
MMseqs2conventional
0.88Not reported0.500.340.61Not reported
MEGAN-LR-ProtBaseline3 sweepsconventional
0.85Not reported0.861.00 (best in column)1.00 (best in column)Not reported
Centrifuge-h5002 sweepsconventional
0.79Not reported0.470.890.53Not reported
MetaMapsconventional
0.78Not reported0.290.300.28Not reported
mOTUsconventional
0.730.320.750.890.890.82
MetaPhlAn3conventional
0.370.730.490.430.310.78
Kraken2Baselineconventional
0.290.340.130.110.110.24
Brackenconventional
0.260.260.120.310.100.17

R recall, P precision.

Caveats and method

What each column measures:

  • PacBio HiFi, ATCC MSA-1003: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 14 classification configurations on PacBio HiFi, ATCC MSA-1003 mock community reads
  • Illumina, ATCC MSA-1003: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 7 classification configurations on Illumina, ATCC MSA-1003 mock community reads
  • PacBio HiFi, Zymo D6331: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 14 classification configurations on PacBio HiFi, Zymo D6331 mock community reads
  • ONT R10.3, Zymo D6300: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 14 classification configurations on ONT R10.3, Zymo D6300 mock community reads
  • ONT Q20, Zymo D6300: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 14 classification configurations on ONT Q20, Zymo D6300 mock community reads
  • Illumina, Zymo D6300: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 7 classification configurations on Illumina, Zymo D6300 mock community reads
  • Mock community of cultured organisms without human host DNA or a clinical specimen matrix.
  • Each method used its own reference database (Table 2), so method and database effects are confounded.
  • Single run per dataset; no intervals.
  • Sourmash rows are reported by sourmash authors (author_reported); the first author is a PacBio employee and MEGAN-LR workflows are PacBio's.
  • Detection uses the authors' threshold of 0.001% of total reads (Methods 'Detection metrics' paragraph 1, Table 3); the authors note it penalises methods that assign fewer reads, and report that a 0.1% threshold cut short-read false positives from several hundred to about 10 or fewer (Results 'Detection metrics' paragraph 5).
  • Two co-authors (C. T. Brown, N. T. Pierce-Ward) wrote the cited sourmash papers and Pierce-Ward ran the analysis; the competing-interests statement does not declare this.
  • Only short-read and general methods were run on the Illumina data.
  • Two community species were absent from several databases and were dropped from species scoring for all methods.
  • The primary ONT datasets exclude reads under 2 kb and over 50 kb; the authors report lower precision and F-scores for all methods when the shorter reads are kept (Results 'Analyses of shorter ONT reads').

Protocols: HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4), Illumina ATCC MSA-1003 mock community (HiSeq 2500): species detection at 0.001% of total reads (Portik et al. 2022 Table 4), HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4), ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4), ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4), Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampled: species detection at 0.001% of total reads (Portik et al. 2022 Table 4)

Primary benchmark evaluation shown (Species detection in mock communities (Portik et al. 2022)). 8 additional study protocols (M. tuberculosis read classification by database (Hall and Coin 2024), Short-read versus host-depleted nanopore mNGS on 62 BALF specimens (Hu et al. 2026), Blood targeted and untargeted sequencing against clinical diagnosis in pneumonia-associated sepsis (Huang et al. 2026) and others) are cited in individual tool drawer rows and under Question, sources and review.

Held for review

Recorded but not shown as evidence until the reason below is resolved.

Viridans streptococci in culture-positive blood and negative controls (Song et al. 2025)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (dna-pathogen-20261009-eval-song2025-bsi-blast-nt-galaxy, dna-pathogen-20261009-eval-song2025-bsi-kraken-native-db-galaxy, dna-pathogen-20261009-eval-song2025-bsi-metaphlan-native-db-galaxy, dna-pathogen-20261009-eval-song2025-bsi-rtg-core-3-12); no evaluation on the protocol is eligible evidence.

Abundance recovery in a log-distributed mock standard (Song et al. 2025)

Withheld: a cited source is unchecked or has evidence concerns; protocol evidence is disputed or unchecked; a reviewed evaluation is no longer eligible (dna-pathogen-20261009-eval-song2025-zymo-blast-nt-galaxy, dna-pathogen-20261009-eval-song2025-zymo-kraken-native-db-galaxy, dna-pathogen-20261009-eval-song2025-zymo-metaphlan-native-db-galaxy, dna-pathogen-20261009-eval-song2025-zymo-rtg-core-3-12); no evaluation on the protocol is eligible evidence.

Not covered yet

  • The composite-reference negative-percent-agreement CI is internally conflicting in the source (54.8-70.0 vs 55.2-70.4) and is excluded from this intake.
  • The exact executable/database version is not pinned in the inspected primary text.
  • The primary article's public redistribution licence is absent; only a compact factual table excerpt is archived.
  • No independent comparison of CZ ID, SURPI, DRAGEN Microbial or BugSeq on clinical DNA specimens with printed per-tool values was found in this pass.
  • Patient-specimen comparisons of workflows (Hu et al. 2026 on BALF, Huang et al. 2026 on plasma) score against references that draw partly on sequencing; only Hu's culture comparison is independent of it. Song et al. 2025 has five culture-positive samples of one organism group and prints relative abundances, not detection calls.
  • Kraken and MetaPhlAn versions and databases are unreported in Song et al. 2025.
  • Per-pipeline culture concordance for 213 sterile body fluids (Lao et al. 2024, nanopore 16S) is printed only in prose and was not extracted.
  • Organisms absent from the reference are covered only indirectly: two species missing from several databases (Portik et al. 2022 Table 4 footnote), database completeness for M. tuberculosis (Hall and Coin 2024), and genus or species exclusion on non-clinical data (the distantly related set of Mock et al. 2022; out-of-database species in Peres da Silva et al. 2024). No clade-exclusion benchmark with a pathogen focus was extracted.
  • LEMMI, CAMI Benchmarking Portal and Govender and Eyre 2022 observed values (figures only) were not used.
  • Song et al. 2025 Table 1 Kraken values are under an evidence concern, so the only patient-sample comparison is withheld.
  • In the linked evidence, machine-learning classifiers are compared with the baseline only on simulated or isolate-derived data, in papers by the developers of BERTax and MetageNN: BERTax and DeepMicrobes against Kraken2 and minimap2 on simulated 1,500-nt genome fragments at superkingdom, phylum and genus rank (Mock et al. 2022), and MetageNN against Kraken2 and MEGAN-LR on ONT reads from 30 pseudo-mock communities of species absent from the reference, where the only printed endpoint is the share of reads assigned at genus level, not whether they were assigned correctly (Peres da Silva et al. 2024). No machine-learning method is compared with the baseline on clinical specimens or at species level.
  • Peres da Silva et al. 2024 show per-tool sensitivity, precision and F1 for MetageNN, Kraken2, MEGAN-LR, MetaMaps and MMseqs2 only in Figures 3 and 4; these were not extracted.

Contribute evidence or propose a correction

How to cite

Select a DNA pathogen-identification workflow for diagnostic testing. Rewire biological model benchmark database, release 2026-11-21-c8bea25e5abb. https://benchmarks.rewirebio.io/use-cases/diagnostic-dna-pathogen-identification/

Release 2026-11-21-c8bea25e5abb. Rankings and values change between releases, so keep the release id in the citation. Cite the original publication as well for any value you reuse: each result links its source.

The compilation and annotations made by Rewire in this release are offered under CC BY 4.0; values and text taken from cited sources stay under the terms of those sources and are not covered by this licence. Data licence

Question, sources and review

Who this is for, setting and exclusions

Inspect the Karius plasma microbial cfDNA positive-percent-agreement evidence against initial blood culture before selecting a workflow and reference standard for the intended specimen population.

Who this is for
  • Clinical researchers scoping the available diagnostic-genomics evidence
  • Computational researchers comparing exact evaluated configurations
Biological setting
The Karius 2019 plasma microbial cfDNA sequencing assay achieves 93.7% (59 of 63) positive percent agreement with initial blood culture in a prospective 350-patient sepsis-alert cohort (348 paired results).
Clinical research scope
Clinical applicability is bounded to agreement with initial blood culture, a reference standard that does not identify every true infection; this is not an estimate of sensitivity against all infections in the cohort.

Outside this use case

  • Sensitivity against a composite/adjudicated reference standard (a separate, excluded endpoint in the source due to a printed CI conflict)
  • RNA viral pathogen detection (this is a DNA-only assay)
Evidence collection plan
Evidence collection planCollecting evidence

Already on record: 9 comparisons shown · 2 held for review · 27 tools, in the evaluated evidence above. The status above describes only the specific comparison in this plan, which remains open; it does not mean no evidence has been collected. The plan defines a comparison to investigate; it does not establish model performance or suitability.

Comparison question

Which DNA sequencing classification workflow detects clinically relevant pathogens and handles contamination or organisms absent from the reference?

Baselines, outcomes and validation requirements

Baselines to include

  • The conventional/author-introduced workflow measured in the linked primary source(s).

Outcomes to measure

  • The declared endpoint in this use case's active mapping(s); see evidence_gaps for what remains open.

Validation requirements

  • Independent held-out population matched to the intended clinical setting.
  • Qualified human scientific review before any clinical-validation claim.

Next collection task

A dedicated rewire-benchmarks protocol/run task with a prospectively adjudicated composite reference standard, not solely initial blood culture.

Sources, review and downloads

Automated source review · 2026-10-07 · Claude Sonnet AMP-integration worker, bounded transcription of Codex-checked primary values; independently reviewed by Codex (workbench/amp-supervision/primary-review.md, integration-review-corrections.md)

Bounded primary-source transcription, independently Codex-checked. No new model execution, independent experimental reproduction, qualified human scientific review or clinical validation.

Release 2026-11-21-c8bea25e5abb · question use-case-diagnostic-dna-pathogen-identification

Use-case input digest bfbb7ab9303becad1e3cb395568c26f41e72ddb67cd1a89a06d9eaa0f76bd95f

Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)