Benchmarking next- versus third-generation sequencing in metagenomics: performance metrics and diagnostic efficacy

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 1 method
• 2 configurations
• 1 benchmark
• 3 protocols
• 1 dataset
• 6 evaluations
• 108 results
• 5 evidence claims

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• Evidence summary: DNA pathogen identification workflows: Summary Most workflow comparisons in the reviewed evidence use mock communities, constructed metagenomes or simulated sequences. Two compare workflows on patient specimens, and in both the reference standard draws partly on sequencing: in Hu et al. 2026 only the culture comparison is independent of the sequencing results, and in Huang et al. 2026 the reference was built mainly from BALF mNGS and the conventional tests that are themselves scored. Both are author-reported and have co-authors from sequencing companies (Genskey Medical Technology for Hu; Geneplus-Beijing, which supplies the targeted assay's probes, for Huang). For the Karius plasma cell-free DNA assay, the developers report 93.7% (59 of 63) agreement with initial blood culture in a sepsis-alert cohort, and blood culture does not detect every infection. A prospective study in children and young adults with high-risk leukaemia, with Karius taking part in sample processing and analysis, found every blood-culture organism in 81.3% (65 of 80) of evaluable bloodstream infection episodes on the day of or after onset and in 51.9% (42 of 81) during the three days before; 82.7% (134 of 162) of control samples from patients without fever or infection had no bacterium or yeast detected. In 62 bronchoalveolar lavage specimens (Hu et al. 2026), a short-read MGI workflow and a host-depleted nanopore workflow both had 95.2% positive agreement with culture but 44.6% and 46.4% negative agreement, because they found organisms culture did not; against a composite reference that adds organisms found by both workflows, positive agreement was 93.3% and 90.7%. In 122 intensive-care patients with pneumonia-associated sepsis (Huang et al. 2026), a targeted probe-capture assay on plasma had 63.46% sensitivity against the reference diagnosis, against 35.58% for untargeted blood mNGS, 49.04% for conventional tests and 9.62% for blood culture, with specificity of 44.44% and 61.11% for the two blood sequencing assays. In mock communities (Portik et al. 2022), Kraken2 and Bracken called 44 to 204 absent species at a 0.001% read threshold (precision 0.05 to 0.21), while long-read MEGAN-LR and BugSeq-V2 called 0 to 2; the first author is a PacBio employee. In seven pathogen-free libraries scored against one shared virus database by the EsViritu developers (preprint), the median number of false virus species was 0 for EsViritu, Centrifuger and Sylph, 6 for Kraken2 and 18 for Metabuli. For M. tuberculosis reads in constructed metagenomes (Hall and Coin 2024), kraken with its standard database classified 0.07 to 0.74 of the M. tuberculosis reads as M. tuberculosis, against 0.97 to 0.995 with a Mycobacterium database. Two papers by the developers of trained classifiers compare them with the baseline tools, on simulated or isolate-derived data rather than clinical specimens. On simulated 1,500-nt genome fragments (Mock et al. 2022, BERTax), Kraken2 and minimap2 assigned a superkingdom to 76.11% and 75.59% of fragments, with 99.82% and 99.95% accuracy on the fragments they assigned, when relatives were in the reference, and to 21.56% and 19.88% with 90.71% and 94.42% accuracy when each fragment's genus was absent from it; BERTax assigned every fragment, with 94.78% and 88.95% accuracy in the two settings. On in silico mixtures of nanopore isolate reads from species absent from the reference (Peres da Silva et al. 2024, MetageNN), MetageNN assigned a genus to 93.26% of reads on average across 30 mixtures, against 64.90% for Kraken2 and 53.06% for MEGAN-LR (on minimap2 alignments); MMseqs2, a reference-based tool not in the baseline, assigned 85.49%. These shares count reads assigned, not reads assigned correctly. Each study used its own reference database and reporting rules, and none of these results is clinical validation. (amp-20261007-dna-pathogens-source: result amp-20261007-dna-pathogens-result; dna-pathogen-20261009-source-portik2022: Table 4; dna-pathogen-20261009-source-hall2024: Tables 5-8; dna-pathogen-20261011-wolf2026-source: Tables 2 and 3; dna-pathogen-20261011-hu2026-source: Table 2; dna-pathogen-20261011-huang2026-source: Table 2; rna-pathogen-20261011-tisza2026-source-supp: Supplemental Table S6; dna-pathogen-20261012-bertax-source: Materials and Methods 'Preparing Data for Fine-Tuning' (1,500-nt fragments; closely and distantly related selection); dna-pathogen-20261012-bertax-source-si: SI Table S2 (superkingdom acc. and prop.); dna-pathogen-20261012-metagenn-source: Results 'Read classification results' paragraph 1 and the pseudo-mock 'Experimental setup' paragraph)
• reference_standard_definition: dna-pathogen-20261011-hu2026-protocol-crs: Reference standard definition The composite reference standard adds to the clinical microbiological tests the organisms that lacked targeted testing results but were detected by both sequencing workflows (MGI and HD-ONT) under the reporting criteria; 31 organisms were added this way. (Methods 'Clinical performance evaluation and reference standards' paragraph 1, third comparator; Results 'Diagnostic performance for clinical samples' paragraph 3)
• Relevance of organism-level agreement with clinical microbiological tests (CMT), 62 BALF specimens (Hu et al. 2026) to "Select a DNA pathogen-identification workflow for diagnostic testing": Assessed by Organism-level agreement with clinical microbiological tests (CMT), 62 BALF specimens (Hu et al. 2026) (dna-pathogen-20261011-hu2026-source: Table 2; dna-pathogen-20261011-hu2026-source-supp: Supplemental Tables S6 and S7)
• Relevance of organism-level agreement with composite reference standard (CRS), 62 BALF specimens (Hu et al. 2026) to "Select a DNA pathogen-identification workflow for diagnostic testing": Assessed by Organism-level agreement with composite reference standard (CRS), 62 BALF specimens (Hu et al. 2026) (dna-pathogen-20261011-hu2026-source: Table 2; dna-pathogen-20261011-hu2026-source-supp: Supplemental Tables S6 and S7)
• Relevance of organism-level agreement with routine culture, 62 BALF specimens (Hu et al. 2026) to "Select a DNA pathogen-identification workflow for diagnostic testing": Assessed by Organism-level agreement with routine culture, 62 BALF specimens (Hu et al. 2026) (dna-pathogen-20261011-hu2026-source: Table 2; dna-pathogen-20261011-hu2026-source-supp: Supplemental Tables S6 and S7)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.