Species detection in mock communities (Portik et al. 2022)
Proxy evidenceDeveloper-reported: Reported by the developers of the evaluated tool.Independent study: Reported in a study whose authors did not develop the evaluated tool.F1 (higher is better), with the other metrics below each value. The best value in each column is highlighted.
| Tool | PacBio HiFi, ATCC MSA-1003 | Illumina, ATCC MSA-1003 | PacBio HiFi, Zymo D6331 | ONT R10.3, Zymo D6300 | ONT Q20, Zymo D6300 | Illumina, Zymo D6300 |
|---|---|---|---|---|---|---|
| 0.93 (best in column) | 0.89 (best in column) | 0.90 | 0.87 | 0.83 | 0.86 | |
BugSeq-V2conventional | 0.89 | Not reported | 0.89 | 0.95 | 0.95 | Not reported |
MMseqs2conventional | 0.88 | Not reported | 0.50 | 0.34 | 0.61 | Not reported |
| 0.85 | Not reported | 0.86 | 1.00 (best in column) | 1.00 (best in column) | Not reported | |
| 0.79 | Not reported | 0.47 | 0.89 | 0.53 | Not reported | |
MetaMapsconventional | 0.78 | Not reported | 0.29 | 0.30 | 0.28 | Not reported |
mOTUsconventional | 0.73 | 0.32 | 0.75 | 0.89 | 0.89 | 0.82 |
MetaPhlAn3conventional | 0.37 | 0.73 | 0.49 | 0.43 | 0.31 | 0.78 |
| 0.29 | 0.34 | 0.13 | 0.11 | 0.11 | 0.24 | |
Brackenconventional | 0.26 | 0.26 | 0.12 | 0.31 | 0.10 | 0.17 |
R recall, P precision.
Caveats and method
What each column measures:
- PacBio HiFi, ATCC MSA-1003: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 14 classification configurations on PacBio HiFi, ATCC MSA-1003 mock community reads
- Illumina, ATCC MSA-1003: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 7 classification configurations on Illumina, ATCC MSA-1003 mock community reads
- PacBio HiFi, Zymo D6331: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 14 classification configurations on PacBio HiFi, Zymo D6331 mock community reads
- ONT R10.3, Zymo D6300: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 14 classification configurations on ONT R10.3, Zymo D6300 mock community reads
- ONT Q20, Zymo D6300: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 14 classification configurations on ONT Q20, Zymo D6300 mock community reads
- Illumina, Zymo D6300: Species-level true and false positives, false negatives, precision, recall, F1, F0.5 and abundance L1 error for 7 classification configurations on Illumina, Zymo D6300 mock community reads
- Mock community of cultured organisms without human host DNA or a clinical specimen matrix.
- Each method used its own reference database (Table 2), so method and database effects are confounded.
- Single run per dataset; no intervals.
- Sourmash rows are reported by sourmash authors (author_reported); the first author is a PacBio employee and MEGAN-LR workflows are PacBio's.
- Detection uses the authors' threshold of 0.001% of total reads (Methods 'Detection metrics' paragraph 1, Table 3); the authors note it penalises methods that assign fewer reads, and report that a 0.1% threshold cut short-read false positives from several hundred to about 10 or fewer (Results 'Detection metrics' paragraph 5).
- Two co-authors (C. T. Brown, N. T. Pierce-Ward) wrote the cited sourmash papers and Pierce-Ward ran the analysis; the competing-interests statement does not declare this.
- Only short-read and general methods were run on the Illumina data.
- Two community species were absent from several databases and were dropped from species scoring for all methods.
- The primary ONT datasets exclude reads under 2 kb and over 50 kb; the authors report lower precision and F-scores for all methods when the shorter reads are kept (Results 'Analyses of shorter ONT reads').
Protocols: HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4), Illumina ATCC MSA-1003 mock community (HiSeq 2500): species detection at 0.001% of total reads (Portik et al. 2022 Table 4), HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4), ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4), ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4), Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampled: species detection at 0.001% of total reads (Portik et al. 2022 Table 4)