Datasets
Challenge-specific metagenomic datasets, including a longitudinal human-gut collection in CAMI III.
Benchmark
CAMI
CAMI is a community benchmark program for metagenomic computational methods.
Sources
cami official source · Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citationsOverview
Metrics
Assembly: genome fraction, NGA50, mismatches, misassemblies and strain precision/recall. Genome binning: purity, completeness, ARI and binned fraction. Taxonomic binning: purity, completeness, F1 and accuracy. Profiling: identification, abundance and diversity metrics, including L1, Bray–Curtis and weighted UniFrac.
Sources
cami2 primary benchmark evidence · Methods: Evaluation metrics; Table 1Allowed inputs
Released sequence data and track-specific reference resources.
Sources
cami official source · Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citationsEvaluation procedure diagram
How it worksEvaluation procedure
Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions.
Sources (2)
cami official source; cami2 primary benchmark evidence · Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1; Methods: Evaluation metrics; Table 1Source reviewed · Automated source review, 2026-09-16. All specifications and missing details
Results
Each comparison retains its reviewed evaluation scope, dataset and metric. Results are shown without a pooled ranking.
CAMI II marine genome binning; pooled short-read gold-standard assembly; circular elements excluded · marmgCAMI2_short_read_pooled_gold_standard_assembly: Accuracy (bp)
Accuracy (bp) (fraction) · Higher values are better.
CAMI II marine genome binning; pooled short-read gold-standard assembly; circular elements excluded · marmgCAMI2_short_read_pooled_gold_standard_assembly · marmgCAMI2_short_read_pooled_gold_standard_assembly
Evidence origin: Independent external evaluation.
CAMI: 501b543f65d62e5c1d6c3813be0badcac5e079ca · line 2; Tool=Gold standard; column Accuracy (bp) through line 17; Tool=Vamb fa045c0 (J1); column Accuracy (bp)- Missing source cells and quarantined conflicts are recorded in acquisition and audit tables. Per-result scoring denominators may be unreported.
Comparison details and limitations
Complete selected source table is retained across source-order panels. These point estimates do not establish statistical significance or a universal ranking.
- Source-specific evaluation. No equivalence to other releases, protocols or model families is inferred.
- Exact source-defined evaluation scope; reported scores are not rewire reproductions.
Automated source review: 2026-09-19. Numerical source review does not establish independent reproduction.
Dots show point estimates. Whiskers show only explicitly defined uncertainty (standard deviation, standard error or a labelled interval); their definitions remain in Table. Unresolved uncertainty is not plotted. Differences do not establish statistical significance.
Showing 12 of 16 matching rows.
Tested configuration
00.250.50.751
Reported scoreMethods and evaluation design
Procedure, tasks and evaluated configurations
How it works
Evaluation methodology
CAMI is a series of blinded metagenomic software challenges. In CAMI II, participants received simulated short and long reads from defined communities and submitted assemblies, genome bins, taxonomic assignments or abundance profiles. Reference truth was used only for scoring, and software versions, input read types and community conditions were kept distinct.
Sources
cami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1Evaluation design
Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.
Tasks
Protocols
These source-backed links do not make different protocols or scores interchangeable.
Baseline coverage
Reference methods help show what a model adds beyond simple controls. We track a null control and a conventional method for each protocol.
0 of 2 active baseline roles have published Rewire measurements in this release. Measurements on a selected protocol do not establish coverage of an entire suite.
Baseline status by linked protocol
Protocol coverage CSV · Model evaluation matrix · Source table · Release and checksums
Coverage is derived from release 2026-09-29-06401fd5b220. Source citations describe the original records; they do not validate an unreviewed baseline proposal. No results have been generated by this audit.
Run this benchmark
Choose a concrete protocol before running an evaluation. Its inputs, split and scoring rules determine which results can be compared.
Run this benchmark
Official portal links task-specific datasets, formats, evaluation software and a submission tutorial. Choose assembly, binning or taxonomic profiling plus a specific data release first; there is no single command for all CAMI tracks. This website snapshot does not pin the linked evaluators.
A maintained rewire runner has not been verified for this benchmark. Check data access, weights, licences, dependencies and hardware in the linked official documentation; requirements have not been fully extracted.
cami official run documentation · Official portal: Resources, Submit, Software and File formats linksStrengths, limitations and unresolved questions
Strengths and limitations
Strengths supported by sources
- Multiple tracks distinguish assembly, binning and abundance estimation.
Sources
cami official source · Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations
Limitations and conditions
- This profile documents CAMI II as a concrete protocol example. Other CAMI rounds may use different genomes, reference databases and metrics; strain diversity and input assembly quality materially change the task.
Sources
cami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1
Profile review details
Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods.
Stable record: discovery-benchmark-camiSpecifications
Inputs, training, access and other details
Explanatory profile: source reviewed · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Datasets | Challenge-specific metagenomic datasets, including a longitudinal human-gut collection in CAMI III.Sourcescami official source · Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations |
| Splits | CAMI II supplied public-genome practice datasets with ground truth before its blinded challenge. Challenge datasets were marine, strain-madness and plant-associated communities; these are challenge conditions, not a standard supervised train/validation/test partition.Sourcescami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 |
| Metrics | Assembly: genome fraction, NGA50, mismatches, misassemblies and strain precision/recall. Genome binning: purity, completeness, ARI and binned fraction. Taxonomic binning: purity, completeness, F1 and accuracy. Profiling: identification, abundance and diversity metrics, including L1, Bray–Curtis and weighted UniFrac.Sourcescami2 primary benchmark evidence · Methods: Evaluation metrics; Table 1 |
| Baselines | Submitted programs are compared under the same data condition. Gold-standard assemblies and MEGAHIT assemblies separate binning performance from upstream assembly error; published method identities and versions are listed in Table 1.Sourcescami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 |
| Leakage controls | Challenge genome data and metadata were kept confidential until the challenge ended. Public reference collections dated 8 January 2019 were supplied for reference-based methods. CAMI II also includes public genomes, so novelty is stratified rather than assumed for every organism.Sourcescami2 primary benchmark evidence · Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 |
| Uncertainty | Uncertainty is task-specific: taxonomic binning Figure 3 uses standard errors across bins; taxonomic profiling Figure 4 reports means across samples with standard deviations. These are not a common seed-based interval for every CAMI metric.Sourcescami2 primary benchmark evidence · Figure 3 and Figure 4 captions: standard error across taxonomic bins versus standard deviation across samples |
| Entity type | Metagenomic community challenge series.Sourcescami official source · Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations |
| Organisms | Microbial communities; CAMI III includes longitudinal human-gut samples.Sourcescami official source · Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations |
| Assays | Challenge-specific metagenomic sequence data and reference composition.Sourcescami official source · Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations |
| Allowed inputs | Released sequence data and track-specific reference resources.Sourcescami official source · Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations |
| Adaptation | Methods process challenge inputs; a challenge edition and track determine resource rules.Sourcescami official source · Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations |
Applicable tests and references
Applicability is distinct from a completed evaluation.
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Papers and result coverage
Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
| Paper or primary resource | Version | Reference |
|---|---|---|
| Critical Assessment of Metagenome Interpretation: the second round of challenges | PMC9007738 | Read source DOI: 10.1038/s41592-022-01431-4 |
Historical gaps recorded on 2026-09-17
The catalogue now holds 256 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.
- Main Table 1 gives best-ranked software, not a complete numerical score table. Assembly, genome binning, taxonomic binning and taxonomic profiling are separate tasks; strain diversity and sample cohorts must be separated. Further source-data extraction remains before new score publication.
Search and extraction details
source found structured extraction pending
Searches
- CAMI II metagenome benchmarking 2022 supplementary results table
Evidence locations
- Table1 rankings; main figures and linked supplementary material
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
21 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims | cami official source Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1; Methods: Evaluation metrics; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Retrieved website snapshot sha256:17825bf33280f40b596a104c547b57fae5ee5c07f8d60b396d0f4780d47ef9a5 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims | cami2 primary benchmark evidence Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1; Methods: Evaluation metrics; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Diagram steps
| cami official source Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1; Methods: Evaluation metrics; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Retrieved website snapshot sha256:17825bf33280f40b596a104c547b57fae5ee5c07f8d60b396d0f4780d47ef9a5 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Diagram steps
| cami2 primary benchmark evidence Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1; Methods: Evaluation metrics; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Evaluation procedure Individual claims | cami official source Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1; Methods: Evaluation metrics; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Retrieved website snapshot sha256:17825bf33280f40b596a104c547b57fae5ee5c07f8d60b396d0f4780d47ef9a5 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Evaluation procedure Individual claims | cami2 primary benchmark evidence Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations; Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1; Methods: Evaluation metrics; Table 1 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Datasets Challenge-specific metagenomic datasets, including a longitudinal human-gut collection in CAMI III. Individual claims | cami official source Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations Version: Retrieved website snapshot sha256:17825bf33280f40b596a104c547b57fae5ee5c07f8d60b396d0f4780d47ef9a5 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits CAMI II supplied public-genome practice datasets with ground truth before its blinded challenge. Challenge datasets were marine, strain-madness and plant-associated communities; these are challenge conditions, not a standard supervised train/validation/test partition. Individual claims | cami2 primary benchmark evidence Methods: Challenge datasets, Challenge organization, Evaluation metrics; Figures 2–4; Table 1 Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Adaptation Methods process challenge inputs; a challenge edition and track determine resource rules. Individual claims | cami official source Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations Version: Retrieved website snapshot sha256:17825bf33280f40b596a104c547b57fae5ee5c07f8d60b396d0f4780d47ef9a5 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Metrics Assembly: genome fraction, NGA50, mismatches, misassemblies and strain precision/recall. Genome binning: purity, completeness, ARI and binned fraction. Taxonomic binning: purity, completeness, F1 and accuracy. Profiling: identification, abundance and diversity metrics, including L1, Bray–Curtis and weighted UniFrac. Individual claims | cami2 primary benchmark evidence Methods: Evaluation metrics; Table 1 Version: PMC9007738 | source checked automated source review · 2026-09-16 Audit detailsPrimary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Independent automated spot review corrected interval terminology and sequence-identity scope against the original figure captions and methods. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
View linked audit checks and correction history
Release 2026-09-29-06401fd5b220 · Record review: discovered
5 source records and release history
- cami official source · Original source · Not reported
- Critical Assessment of Metagenome Interpretation: the second round of challenges · Original source · PMC9007738
- cami official source · Original source · Retrieved website snapshot sha256:17825bf33280f40b596a104c547b57fae5ee5c07f8d60b396d0f4780d47ef9a5
- cami official run documentation · Original source · Official documentation snapshot 2026-09-17; mutable website
- cami2 primary benchmark evidence · Original source · PMC9007738
Technical metadata and extraction receipts
Stable ID: discovery-benchmark-cami
- areas
- microbiome
- entity level
- challenge
- scope note
- Specialist molecular or omics evaluation; protocol details require review before numerical comparison.
- task
- Metagenomic assembly, binning and profiling assessment
- version
- Not reported
- benchmark research
- review date: 2026-09-17; status: source_found_structured_extraction_pending; primary sources: evidence-expansion-p2-evidence-discovery-final-cami2-da932c1cde8b; inspected locators: Table1 rankings; main figures and linked supplementary material; searched queries: CAMI II metagenome benchmarking 2022 supplementary results table; gaps: Main Table 1 gives best-ranked software, not a complete numerical score table. Assembly, genome binning, taxonomic binning and taxonomic profiling are separate tasks; strain diversity and sample cohorts must be separated. Further source-data extraction remains before new score publication.; claim scope: Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
- historical missing metadata
- dataset release: unextracted; metric implementation: unextracted; split manifest: unextracted; version: unextracted
- metadata review scope
- historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
- entity classification
- review date: 2026-09-17; rationale: The cited profile describes an organized challenge with category- or round-specific evaluation rules; retain it as a top-level benchmark, without conflating different editions or protocols.; source ids: evidence-benchmark-cami-snapshot; source locator: Official CAMI homepage: initiative description; challenges; dataset correction notices; toolkit citations; ambiguities: None recorded
- run documentation
- record id: discovery-benchmark-cami; source ids: run-doc-cami-official-20260917; status: official_documentation_linked; summary: Official portal links task-specific datasets, formats, evaluation software and a submission tutorial. Choose assembly, binning or taxonomic profiling plus a specific data release first; there is no single command for all CAMI tracks. This website snapshot does not pin the linked evaluators.; source locator: Official portal: Resources, Submit, Software and File formats links
Related records
- part of: CAMI II marine genome binning; pooled short-read gold-standard assembly; circular elements excluded · marmgCAMI2_short_read_pooled_gold_standard_assembly
- parent: CAMI genome binning
- part of: CAMI genome binning
- parent: CAMI metagenome assembly
- part of: CAMI metagenome assembly
- parent: CAMI taxonomic binning
- part of: CAMI taxonomic binning
- parent: CAMI taxonomic profiling
- part of: CAMI taxonomic profiling
- applicable to: CAMISIM
- applicable to: Kraken 2
- applicable to: MetaPhlAn