Datasets
Synthetic and simulated communities with known reference composition.
Task
Long-read taxonomic profiling
Long-read taxonomic profiling measures both taxon detection and relative-abundance agreement.
Sources
Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passagesOverview
Metrics
Genus/species precision, recall and F1 after abundance thresholding; normalized L1 loss or Spearman correlation for abundance.
Allowed inputs
Long sequencing reads.
Sources
Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passagesEvaluation procedure diagram
How it worksComputational evaluation flow
Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.
Sources
Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages; Methods: Synthetic and simulated datasetsSource reviewed · Automated source review, 2026-09-16. All specifications and missing details
Results
Each comparison retains its reviewed evaluation scope, dataset and metric. Results are shown without a pooled ranking.
Species profiling on Dilthey2019 simulated long reads · Table 1:
Recall (fraction) · Higher values are better.
Species profiling on Dilthey2019 simulated long reads (Long-read taxonomic profiling) · Species profiling on Dilthey2019 simulated long reads
Evidence origin: Author-reported evaluation, Independent external evaluation, Result quoted from another source.
Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Table 1:: Recall, Species profiling on Dilthey2019 simulated long reads- MetaMaps is quoted prior evidence, not an independent repeated run.
- Reference databases and classifier-versus-profiler task differ.
Comparison details and limitations
Species-level recall/precision/F 1. MetaMaps results copied from original manuscript; other tool versions reported in Methods. 200,114 reads;96 strain design,94 strains with RefSeq species representatives; average read 4,997 bp.
- No interval assigned unless printed in source cell.
Automated source review: 2026-09-17. Numerical source review does not establish independent reproduction.
Dots show point estimates. Whiskers show only explicitly defined uncertainty (standard deviation, standard error or a labelled interval); their definitions remain in Table. Unresolved uncertainty is not plotted. Differences do not establish statistical significance.
Showing 7 of 7 matching rows.
Methods and evaluation design
Procedure, tasks and evaluated configurations
How it works
Evaluation methodology
Synthetic and simulated communities with known reference composition. Known-composition simulated or mock-community samples are profiled against reference resources. The Dilthey simulation has RefSeq species representatives for 94 of 96 strains; the additional simulation selects RefSeq-represented species. Genus/species precision, recall and F1 after abundance thresholding; normalized L1 loss or Spearman correlation for abundance. Lemur v1.0.1 is evaluated alone and with Magnet against Centrifuger v1.0.0, Kraken 2 v2.1.3, Melon v0.1.0, MetaMaps commit 633d2e0 and Sourmash v4.8.2. Melon lacks fungal references; the paper also reports bacterial-only comparisons for fungal-containing datasets. Reference availability is part of this identification task: 94 of 96 strains in the first simulation have a corresponding RefSeq species representative. The additional metagenome simulation deliberately selects species with RefSeq representative genomes and available MAGs. These settings do not establish novel-species generalization.
Sources
Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages; Methods: Synthetic and simulated datasets; Methods: Method Comparison; Methods: Simulated data from Dilthey et al. 2019; Simulated metagenomeEvaluation design
Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.
These source-backed links do not make different protocols or scores interchangeable.
Recorded evaluations
Each evaluation records what was tested and under which conditions.
Run this benchmark
Choose a concrete protocol before running an evaluation. Its inputs, split and scoring rules determine which results can be compared.
Run instructions
No runnable recipe has been reviewed for this task. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.
A task describes a biological question. Choose a linked protocol to obtain concrete split and scoring instructions.
Strengths, limitations and unresolved questions
Strengths and limitations
Strengths supported by sources
No source-reviewed explanatory claims are recorded here yet.
Limitations and conditions
- Spearman is computed only over taxa present in both truth and prediction, so it omits false-positive and false-negative taxa. Presence/absence thresholds vary for the Zymo EVEN setting.
Sources
Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages
Profile review details
Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.
Stable record: reported-task-6330d593980b5bSpecifications
Inputs, training, access and other details
Explanatory profile: source reviewed · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Datasets | Synthetic and simulated communities with known reference composition.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages |
| Splits | Known-composition simulated or mock-community samples are profiled against reference resources. The Dilthey simulation has RefSeq species representatives for 94 of 96 strains; the additional simulation selects RefSeq-represented species.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets |
| Metrics | Genus/species precision, recall and F1 after abundance thresholding; normalized L1 loss or Spearman correlation for abundance.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages |
| Baselines | Lemur v1.0.1 is evaluated alone and with Magnet against Centrifuger v1.0.0, Kraken 2 v2.1.3, Melon v0.1.0, MetaMaps commit 633d2e0 and Sourmash v4.8.2. Melon lacks fungal references; the paper also reports bacterial-only comparisons for fungal-containing datasets.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Method Comparison |
| Leakage controls | Reference availability is part of this identification task: 94 of 96 strains in the first simulation have a corresponding RefSeq species representative. The additional metagenome simulation deliberately selects species with RefSeq representative genomes and available MAGs. These settings do not establish novel-species generalization.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Simulated data from Dilthey et al. 2019; Simulated metagenome |
| Uncertainty | Mean and standard deviation across five replicate runs are reported for the Zymo EVEN and LOG comparisons; read subsampling also uses repeated seeds.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages |
| Entity type | Paper-specific computational evaluation protocol.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages |
| Organisms | Synthetic and simulated microbial communities.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages |
| Assays | Known taxonomic composition of sequence mixtures.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages |
| Allowed inputs | Long sequencing reads.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages |
| Adaptation | Lemur and the comparator profilers use their reference resources; Magnet additionally aligns reads to cluster-representative genomes with minimap2. This is reference-based taxonomic profiling, rather than fitting a classifier on labeled train/test folds.SourcesLightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Methods: Competitive read alignment with Magnet; Method Comparison |
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Papers and result coverage
Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
| Paper or primary resource | Version | Reference |
|---|---|---|
| Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet | PMC archival version PMC11185576.2 | Read source DOI: 10.1101/2024.06.01.596961 |
Historical gaps recorded on 2026-09-17
The catalogue now holds 23 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.
- Independent batch review before import; preserve existing observation identities.
Search and extraction details
complete comparison tables extracted pending publication review
Searches
- Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet primary paper benchmark results
Evidence locations
- Tables1–3; simulated Dilthey2019 and ZymoEVEN/LOG cohorts
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
17 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Diagram caption Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol. Individual claims | Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages; Methods: Synthetic and simulated datasets Version: PMC archival version PMC11185576.2 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Diagram steps
| Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages; Methods: Synthetic and simulated datasets Version: PMC archival version PMC11185576.2 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Computational evaluation flow Individual claims | Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages; Methods: Synthetic and simulated datasets Version: PMC archival version PMC11185576.2 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Datasets Synthetic and simulated communities with known reference composition. Individual claims | Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages Version: PMC archival version PMC11185576.2 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits Known-composition simulated or mock-community samples are profiled against reference resources. The Dilthey simulation has RefSeq species representatives for 94 of 96 strains; the additional simulation selects RefSeq-represented species. Individual claims | Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet Methods: Synthetic and simulated datasets Version: PMC archival version PMC11185576.2 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Adaptation Lemur and the comparator profilers use their reference resources; Magnet additionally aligns reads to cluster-representative genomes with minimap2. This is reference-based taxonomic profiling, rather than fitting a classifier on labeled train/test folds. Individual claims | Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet Methods: Competitive read alignment with Magnet; Method Comparison Version: PMC archival version PMC11185576.2 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Metrics Genus/species precision, recall and F1 after abundance thresholding; normalized L1 loss or Spearman correlation for abundance. Individual claims | Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages Version: PMC archival version PMC11185576.2 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Baselines Lemur v1.0.1 is evaluated alone and with Magnet against Centrifuger v1.0.0, Kraken 2 v2.1.3, Melon v0.1.0, MetaMaps commit 633d2e0 and Sourmash v4.8.2. Melon lacks fungal references; the paper also reports bacterial-only comparisons for fungal-containing datasets. Individual claims | Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet Methods: Method Comparison Version: PMC archival version PMC11185576.2 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Leakage controls Reference availability is part of this identification task: 94 of 96 strains in the first simulation have a corresponding RefSeq species representative. The additional metagenome simulation deliberately selects species with RefSeq representative genomes and available MAGs. These settings do not establish novel-species generalization. Individual claims | Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet Methods: Simulated data from Dilthey et al. 2019; Simulated metagenome Version: PMC archival version PMC11185576.2 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Uncertainty Mean and standard deviation across five replicate runs are reported for the Zymo EVEN and LOG comparisons; read subsampling also uses repeated seeds. Individual claims | Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages Version: PMC archival version PMC11185576.2 | source checked automated source review · 2026-09-16 Audit detailsTargeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
View linked audit checks and correction history
Release 2026-09-29-06401fd5b220 · Record review: needs review
2 source records and release history
- Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Original source · PMC archival version PMC11185576.2
- Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet · Original source · PMC archival version PMC11185576.2
Technical metadata and extraction receipts
Stable ID: reported-task-6330d593980b5b
- areas
- microbes-communities
- tasks
- Long-read taxonomic profiling
- entity level
- task
- version
- Not reported
- task
- Long-read taxonomic profiling
- scope note
- Paper-specific evaluation task; protocol completeness requires further extraction.
- comparison panels
- id: part2-lemur-magnet-2024-T1-a3172c2928; title: Species profiling on Dilthey2019 simulated long reads · Table 1:; protocol id: paper-protocol-1080592b8d6a8ce0bd; dataset id: paper-dataset-e8eb310c5d40e647ac; metric: Recall; unit: fraction; direction: higher; result ids: paper-result-80f8cca2c2d9379a1a; paper-result-620e5294f3d001ac96; paper-result-a91cd2aadeccebf9b9; paper-result-b70bf4b4c41e6dd812; paper-result-6da2d59412e925bccd; paper-result-74d2e70b442ba9026b; paper-result-8fd5837795f2fbba5f; source ids: part2-lemur-magnet-2024; source locator: Table 1:: Recall, Species profiling on Dilthey2019 simulated long reads; context: Species-level recall/precision/F 1. MetaMaps results copied from original manuscript; other tool versions reported in Methods. 200,114 reads;96 strain design,94 strains with RefSeq species representatives; average read 4,997 bp.; caveats: MetaMaps is quoted prior evidence, not an independent repeated run.; Reference databases and classifier-versus-profiler task differ.; No interval assigned unless printed in source cell.; review: method: automated_source_review; date: 2026-09-17; id: part2-lemur-magnet-2024-T1-81ab2d846d; title: Species profiling on Dilthey2019 simulated long reads · Table 1:; protocol id: paper-protocol-1080592b8d6a8ce0bd; dataset id: paper-dataset-e8eb310c5d40e647ac; metric: Precision; unit: fraction; direction: higher; result ids: paper-result-2268ad4341b9316164; paper-result-06a95cb8e4f61822eb; paper-result-9a39bac49d7dcf2b79; paper-result-a6256bb040cd29f64f; paper-result-f53c61bd32688408d8; paper-result-e2ceeb0f245677c087; paper-result-0dc829636e77fc0d92; source ids: part2-lemur-magnet-2024; source locator: Table 1:: Precision, Species profiling on Dilthey2019 simulated long reads; context: Species-level recall/precision/F 1. MetaMaps results copied from original manuscript; other tool versions reported in Methods. 200,114 reads;96 strain design,94 strains with RefSeq species representatives; average read 4,997 bp.; caveats: MetaMaps is quoted prior evidence, not an independent repeated run.; Reference databases and classifier-versus-profiler task differ.; No interval assigned unless printed in source cell.; review: method: automated_source_review; date: 2026-09-17; id: part2-lemur-magnet-2024-T1-e9572b0ed9; title: Species profiling on Dilthey2019 simulated long reads · Table 1:; protocol id: paper-protocol-1080592b8d6a8ce0bd; dataset id: paper-dataset-e8eb310c5d40e647ac; metric: F1 score; unit: fraction; direction: higher; result ids: paper-result-4468fa08040fb0bd80; paper-result-19f79487e8dff1a781; paper-result-25455d4621f0aeb383; paper-result-f9b8d072704477374e; paper-result-03860ee74178c46356; paper-result-fdc2db2e65a30e98e7; paper-result-f8adf74d2b525a6380; source ids: part2-lemur-magnet-2024; source locator: Table 1:: F1 score, Species profiling on Dilthey2019 simulated long reads; context: Species-level recall/precision/F 1. MetaMaps results copied from original manuscript; other tool versions reported in Methods. 200,114 reads;96 strain design,94 strains with RefSeq species representatives; average read 4,997 bp.; caveats: MetaMaps is quoted prior evidence, not an independent repeated run.; Reference databases and classifier-versus-profiler task differ.; No interval assigned unless printed in source cell.; review: method: automated_source_review; date: 2026-09-17
- benchmark research
- review date: 2026-09-17; status: complete_comparison_tables_extracted_pending_publication_review; primary sources: part2-lemur-magnet-2024; inspected locators: Tables1–3; simulated Dilthey2019 and ZymoEVEN/LOG cohorts; searched queries: Lightweight taxonomic profiling of long-read metagenomic datasets with Lemur and Magnet primary paper benchmark results; gaps: Independent batch review before import; preserve existing observation identities.; claim scope: Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
- historical missing metadata
- protocol version: not_reported_in_legacy_extract; split: not_reported_in_legacy_extract
- metadata review scope
- historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
- legacy kinds
- benchmark
- entity classification
- review date: 2026-09-17; rationale: This source-scoped record identifies the biological prediction task and holds its paper context. Preserve the existing task identity; exact split, model adaptation and scoring remain in linked evaluations or separate protocol records.; source ids: lemur-magnet-2024; source locator: Methods: Synthetic and simulated datasets; Method evaluation; cached text lines 75–77, 97–99; uncertainty/repeat-run/statistical-comparison passages; ambiguities: A paper- or suite-specific task may constrain some inputs or metrics; that alone does not make it interchangeable with a complete versioned protocol. No protocol equivalence is inferred.; Some legacy profile Entity type facts use the generic phrase computational evaluation protocol. That boilerplate is not sufficient to establish a single fixed protocol identity or to merge this task with another protocol record.
Related records
- dataset: Zymo LOG 10%
- benchmark: Lemur: Long-read taxonomic profiling
- benchmark: Kraken 2: Long-read taxonomic profiling
- evaluates task: Species profiling on Dilthey2019 simulated long reads (Long-read taxonomic profiling)