rewirebio.iobenchmarks
Contested result

Centrifuge and Genome Detective: de Vries et al. medRxiv preprint v1, Supplementary Tables 2-4 and Carbo et al. medRxiv preprint v1, Supplementary Tables 1 and 2 disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

de Vries et al. medRxiv preprint v1, Supplementary Tables 2-4 ranks Centrifuge above Genome Detective in 5 of 5 comparisons; Carbo et al. medRxiv preprint v1, Supplementary Tables 1 and 2 ranks Genome Detective above Centrifuge in 12 of 13 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank Centrifuge above Genome Detective in 6 of their 35 comparisons and Genome Detective higher in 12; 17 are ties. Of those papers, Carbo et al. 2022 ranks Genome Detective higher on its own (12 of 28).

Papers grouped by which system they rank higher: Centrifuge or Genome DetectiveRank Centrifuge higher: de Vries et al. medRxiv preprint v1…. Rank Genome Detective higher: Carbo et al. 2022. Mixed: none.Rank Centrifuge higherMixedRank Genome Detective higherde Vries et al.medRxiv preprint v1…de Vries et al. medRxiv preprint v1, Supplementary Tables 2-4Carbo et al. 2022Carbo et al. medRxiv preprint v1, Supplementary Tables 1 and 2
  • Rank Centrifuge higher (1): de Vries et al. medRxiv preprint v1…
  • Rank Genome Detective higher (1): Carbo et al. 2022
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Centrifuge or Genome Detective higherCarbo et al. 2022: Centrifuge higher in 1, Genome Detective higher in 12, 15 ties. de Vries et al. medRxiv preprint v1…: Centrifuge higher in 5, Genome Detective higher in 0, 2 ties.Carbo et al. 2022Carbo et al. medRxiv preprint v1, Supplementary Tables 1 and 21 Centrifuge higher15 tie1512 Genome Detective higher12de Vries et al. medRxiv preprint v1…de Vries et al. medRxiv preprint v1, Supplementary Tables 2-45 Centrifuge higher52 tie2
Centrifuge higherTieGenome Detective higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks Centrifuge higherRanks Genome Detective higherTies
Carbo et al. medRxiv preprint v1, Supplementary Tables 1 and 2 · DOIAgrees11215
de Vries et al. medRxiv preprint v1, Supplementary Tables 2-4 · DOIAgrees502
Comparisons in Carbo et al. 2022 (28)
ProtocolDatasetMetricCentrifugeGenome DetectiveHigher
Respiratory virus detection against PCR, family level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)AUROC (ROC over read-count cut-offs), unitless0.9410.961Genome Detective
Respiratory virus detection against PCR, family level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Negative predictive value (at the ROC-selected read-count cut-off), fraction0.9990.999Tie
Respiratory virus detection against PCR, family level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Pearson R² (assigned read count against PCR Ct value), percent34.843.4Genome Detective
Respiratory virus detection against PCR, family level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Precision (at the ROC-selected read-count cut-off), fraction0.3540.469Genome Detective
Respiratory virus detection against PCR, family level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Recall (at the ROC-selected read-count cut-off), fraction0.9580.958Tie
Respiratory virus detection against PCR, family level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)ROC distance, unitless0.06880.0537Genome Detective
Respiratory virus detection against PCR, family level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Specificity (at the ROC-selected read-count cut-off; printed as selectivity (SL)), fraction0.9450.966Genome Detective
Respiratory virus detection against PCR, genus level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)AUROC (ROC over read-count cut-offs), unitless0.9410.948Tie
Respiratory virus detection against PCR, genus level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Negative predictive value (at the ROC-selected read-count cut-off), fraction0.9970.997Tie
Respiratory virus detection against PCR, genus level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Pearson R² (assigned read count against PCR Ct value), percent33.242.3Genome Detective
Respiratory virus detection against PCR, genus level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Precision (at the ROC-selected read-count cut-off), fraction0.5640.611Genome Detective
Respiratory virus detection against PCR, genus level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Recall (at the ROC-selected read-count cut-off), fraction0.9170.917Tie
Respiratory virus detection against PCR, genus level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)ROC distance, unitless0.08620.0853Genome Detective
Respiratory virus detection against PCR, genus level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Specificity (at the ROC-selected read-count cut-off; printed as selectivity (SL)), fraction0.9780.982Tie
Respiratory virus detection against PCR, species level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)AUROC (ROC over read-count cut-offs), unitless0.9430.951Tie
Respiratory virus detection against PCR, species level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Negative predictive value (at the ROC-selected read-count cut-off), fraction0.9970.997Tie
Respiratory virus detection against PCR, species level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Pearson R² (assigned read count against PCR Ct value), percent32.242.3Genome Detective
Respiratory virus detection against PCR, species level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Precision (at the ROC-selected read-count cut-off), fraction0.5790.667Genome Detective
Respiratory virus detection against PCR, species level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Recall (at the ROC-selected read-count cut-off), fraction0.9170.917Tie
Respiratory virus detection against PCR, species level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)ROC distance, unitless0.08590.0846Genome Detective
Respiratory virus detection against PCR, species level, read-count cut-off 0 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Specificity (at the ROC-selected read-count cut-off; printed as selectivity (SL)), fraction0.9790.986Tie
Respiratory virus detection against PCR, species level, read-count cut-off 10 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)AUROC (ROC over read-count cut-offs), unitless0.9350.934Tie
Respiratory virus detection against PCR, species level, read-count cut-off 10 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Negative predictive value (at the ROC-selected read-count cut-off), fraction0.9960.996Tie
Respiratory virus detection against PCR, species level, read-count cut-off 10 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Pearson R² (assigned read count against PCR Ct value), percent14.235.6Genome Detective
Respiratory virus detection against PCR, species level, read-count cut-off 10 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Precision (at the ROC-selected read-count cut-off), fraction0.8750.808Centrifuge
Respiratory virus detection against PCR, species level, read-count cut-off 10 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Recall (at the ROC-selected read-count cut-off), fraction0.8750.875Tie
Respiratory virus detection against PCR, species level, read-count cut-off 10 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)ROC distance, unitless0.1250.125Tie
Respiratory virus detection against PCR, species level, read-count cut-off 10 (Carbo et al. 2022 Supplementary Table 1)Nasal washings from COPD patients with respiratory complaints, metagenomic sequencing and 13-target respiratory PCR panel (Carbo et al. 2022)Specificity (at the ROC-selected read-count cut-off; printed as selectivity (SL)), fraction0.9960.994Tie
Comparisons in de Vries et al. medRxiv preprint v1… (7)
ProtocolDatasetMetricCentrifugeGenome DetectiveHigher
Virus hits against RT-PCR: true and false positives, PPV and hit-level sensitivity (de Vries et al. 2021 Supplementary Table 4)ENNGS benchmark: 13 clinical metagenomic datasets with RT-PCR results (de Vries et al. 2021)False negatives (PCR-positive hits not reported), count12Centrifuge
Virus hits against RT-PCR: true and false positives, PPV and hit-level sensitivity (de Vries et al. 2021 Supplementary Table 4)ENNGS benchmark: 13 clinical metagenomic datasets with RT-PCR results (de Vries et al. 2021)False positives (additional viral hits with a negative PCR result), count00Tie
Virus hits against RT-PCR: true and false positives, PPV and hit-level sensitivity (de Vries et al. 2021 Supplementary Table 4)ENNGS benchmark: 13 clinical metagenomic datasets with RT-PCR results (de Vries et al. 2021)Precision (positive predictive value, hits with PCR data), percent100100Tie
Virus hits against RT-PCR: true and false positives, PPV and hit-level sensitivity (de Vries et al. 2021 Supplementary Table 4)ENNGS benchmark: 13 clinical metagenomic datasets with RT-PCR results (de Vries et al. 2021)Recall (hit level, 15 PCR-positive hits), percent93.386.7Centrifuge
Virus hits against RT-PCR: true and false positives, PPV and hit-level sensitivity (de Vries et al. 2021 Supplementary Table 4)ENNGS benchmark: 13 clinical metagenomic datasets with RT-PCR results (de Vries et al. 2021)True positives (hits with a positive PCR result, out of 15), count1413Centrifuge
Detection of RT-PCR-positive viruses per sample, with assigned read counts (de Vries et al. 2021 Supplementary Table 2)ENNGS benchmark: 13 clinical metagenomic datasets with RT-PCR results (de Vries et al. 2021)Recall (sample level, 13 samples), percent92.384.6Centrifuge
Detection of RT-PCR-positive viruses per sample, with assigned read counts (de Vries et al. 2021 Supplementary Table 2)ENNGS benchmark: 13 clinical metagenomic datasets with RT-PCR results (de Vries et al. 2021)True positives (samples correctly positive out of 13, mixed infections counted once), count1211Centrifuge

What disagrees

studies
2
Compared
Centrifuge, Genome Detective
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-c332f3dd95f7d5e3 · All contested results · How findings are made and followed up