rewirebio.iobenchmarks
Contested result

Kraken2 and sourmash: Taxonomic classification of DNA sequences beyond sequence similarity using deep neural networks (PNAS PDF) and Evaluation of taxonomic classification and profiling methods for long-read shotgun metagenomic sequencing datasets disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

Taxonomic classification of DNA sequences beyond sequence similarity using deep neural networks (PNAS PDF) ranks Kraken2 above sourmash in 2 of 2 comparisons; Evaluation of taxonomic classification and profiling methods for long-read shotgun metagenomic sequencing datasets ranks sourmash above Kraken2 in 24 of 29 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank Kraken2 above sourmash in 7 of their 52 comparisons and sourmash higher in 24; 21 are ties. Of those papers, Evaluation of taxonomic classification… ranks sourmash higher on its own (24 of 48).

Papers grouped by which system they rank higher: Kraken2 or sourmashRank Kraken2 higher: Taxonomic classification of DNA…. Rank sourmash higher: Evaluation of taxonomic classification…. Mixed: none.Rank Kraken2 higherMixedRank sourmash higherTaxonomicclassification of DNA…Taxonomic classification of DNA sequences beyond sequence similarity using deep neural networks (PNAS PDF)Evaluation of taxonomicclassification…Evaluation of taxonomic classification and profiling methods for long-read shotgun metagenomic sequencing datasets
  • Rank Kraken2 higher (1): Taxonomic classification of DNA…
  • Rank sourmash higher (1): Evaluation of taxonomic classification…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Kraken2 or sourmash higherEvaluation of taxonomic classification…: Kraken2 higher in 5, sourmash higher in 24, 19 ties. Taxonomic classification of DNA…: Kraken2 higher in 2, sourmash higher in 0, 2 ties.Evaluation of taxonomic classification…Evaluation of taxonomic classification and profiling methods for long-read shotgun metagenomic sequencing datasets5 Kraken2 higher519 tie1924 sourmash higher24Taxonomic classification of DNA…Taxonomic classification of DNA sequences beyond sequence similarity using deep neural networks (PNAS PDF)2 Kraken2 higher22 tie2
Kraken2 higherTiesourmash higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Evaluation of taxonomic classification… (48)
ProtocolDatasetMetricKraken2sourmashHigher
HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi)Composition L1 distance (species-level relative abundance, false positives pooled as 'Other'), percent50.768.6Kraken2
HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi)F-beta (F0.5 (beta = 0.5)), fraction0.210.83sourmash
HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi)F1, fraction0.290.89sourmash
HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi)False negatives, count00Tie
HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi)False positives, count965sourmash
HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi)Precision, fraction0.170.8sourmash
HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi)Recall, fraction11Tie
HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ATCC MSA-1003 mock community (PacBio Sequel II HiFi)True positives, count2020Tie
HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi)Composition L1 distance (species-level relative abundance, false positives pooled as 'Other'), percent51.952Tie
HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi)F-beta (F0.5 (beta = 0.5)), fraction0.080.93sourmash
HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi)F1, fraction0.130.93sourmash
HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi)False negatives, count11Tie
HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi)False positives, count1961sourmash
HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi)Precision, fraction0.070.93sourmash
HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi)Recall, fraction0.930.93Tie
HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)HiFi ZymoBIOMICS D6331 gut microbiome standard (PacBio Sequel II HiFi)True positives, count1414Tie
Illumina ATCC MSA-1003 mock community (HiSeq 2500): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ATCC MSA-1003 mock community (HiSeq 2500)Composition L1 distance (species-level relative abundance, false positives pooled as 'Other'), percent44.857.2Kraken2
Illumina ATCC MSA-1003 mock community (HiSeq 2500): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ATCC MSA-1003 mock community (HiSeq 2500)F-beta (F0.5 (beta = 0.5)), fraction0.240.78sourmash
Illumina ATCC MSA-1003 mock community (HiSeq 2500): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ATCC MSA-1003 mock community (HiSeq 2500)F1, fraction0.340.85sourmash
Illumina ATCC MSA-1003 mock community (HiSeq 2500): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ATCC MSA-1003 mock community (HiSeq 2500)False negatives, count00Tie
Illumina ATCC MSA-1003 mock community (HiSeq 2500): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ATCC MSA-1003 mock community (HiSeq 2500)False positives, count777sourmash
Illumina ATCC MSA-1003 mock community (HiSeq 2500): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ATCC MSA-1003 mock community (HiSeq 2500)Precision, fraction0.210.74sourmash
Illumina ATCC MSA-1003 mock community (HiSeq 2500): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ATCC MSA-1003 mock community (HiSeq 2500)Recall, fraction11Tie
Illumina ATCC MSA-1003 mock community (HiSeq 2500): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ATCC MSA-1003 mock community (HiSeq 2500)True positives, count2020Tie
Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampled: species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampledComposition L1 distance (species-level relative abundance, false positives pooled as 'Other'), percent59.799.3Kraken2
Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampled: species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampledF-beta (F0.5 (beta = 0.5)), fraction0.170.86sourmash
Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampled: species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampledF1, fraction0.240.91sourmash
Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampled: species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampledFalse negatives, count00Tie
Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampled: species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampledFalse positives, count622sourmash
Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampled: species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampledPrecision, fraction0.140.83sourmash
Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampled: species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampledRecall, fraction11Tie
Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampled: species detection at 0.001% of total reads (Portik et al. 2022 Table 4)Illumina ZymoBIOMICS D6300 standard (NovaSeq 6000), subsampledTrue positives, count1010Tie
ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION)Composition L1 distance (species-level relative abundance, false positives pooled as 'Other'), percent16.755.3Kraken2
ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION)F-beta (F0.5 (beta = 0.5)), fraction0.070.76sourmash
ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION)F1, fraction0.110.83sourmash
ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION)False negatives, count00Tie
ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION)False positives, count1664sourmash
ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION)Precision, fraction0.060.71sourmash
ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION)Recall, fraction11Tie
ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT Q20 ZymoBIOMICS D6300 standard, length-filtered (PromethION)True positives, count1010Tie
ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION)Composition L1 distance (species-level relative abundance, false positives pooled as 'Other'), percent22.347.7Kraken2
ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION)F-beta (F0.5 (beta = 0.5)), fraction0.070.53sourmash
ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION)F1, fraction0.110.64sourmash
ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION)False negatives, count00Tie
ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION)False positives, count15611sourmash
ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION)Precision, fraction0.060.47sourmash
ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION)Recall, fraction11Tie
ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION): species detection at 0.001% of total reads (Portik et al. 2022 Table 4)ONT R10.3 ZymoBIOMICS D6300 standard, length-filtered (GridION)True positives, count1010Tie
Comparisons in Taxonomic classification of DNA… (4)
ProtocolDatasetMetricKraken2sourmashHigher
BERTax Closely related dataset: taxonomic classification at superkingdom and phylum (Mock et al. 2022)BERTax Closely related dataset (Mock et al. 2022)Accuracy (phylum rank; among the samples the tool classified; SI Table S2 'acc.'), percent98.899.8Tie
BERTax Closely related dataset: taxonomic classification at superkingdom and phylum (Mock et al. 2022)BERTax Closely related dataset (Mock et al. 2022)Accuracy (superkingdom rank; among the samples the tool classified; SI Table S2 'acc.'), percent99.8100Tie
BERTax Closely related dataset: taxonomic classification at superkingdom and phylum (Mock et al. 2022)BERTax Closely related dataset (Mock et al. 2022)Macro average precision (phylum rank; macro average over classes; unclassified samples treated as random guesses), percent769.04Kraken2
BERTax Closely related dataset: taxonomic classification at superkingdom and phylum (Mock et al. 2022)BERTax Closely related dataset (Mock et al. 2022)Macro average precision (superkingdom rank; macro average over classes; unclassified samples treated as random guesses), percent86.330.7Kraken2

What disagrees

studies
2
Compared
Kraken2, sourmash
Study that disagrees
Other studies
Rule
Two-study conflict; different-task
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-847650b9e481c666 · All contested results · How findings are made and followed up