BacBench repository README at commit 64b7d45
Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa · Project source
What this source contributed
Records in this release that cite this source. A record can cite several sources.
Claims
Statements this source supports, each with where in the source it was checked.
- Apache-2.0 covers the BacBench code: Licence Apache License 2.0, stated by the README licence badge and carried by the LICENSE file at the pinned commit (BacBench README licence badge, line 3; LICENSE file at the pinned commit)
- BacBench changed the essential gene and PPI splits on 2026-05-13: Protocol change On 13 May 2026 BacBench updated the essential gene and protein-protein interaction tasks to a phylogeny-aware split by genus, added BacLM as a benchmarked model and deprecated the strain clustering task (BacBench README 'News', entry dated 13.05.2026)
- BacBench has five active tasks and one deprecated task: Tasks Five active tasks (essential gene prediction, operon identification from long-read RNA sequencing, protein-protein interaction, antibiotic resistance prediction and phenotypic trait prediction) and one deprecated task (strain clustering), which remains in the repository for reproducibility (BacBench README opening paragraph and 'Task overview' table)
- BacBench publishes no numeric results: Reported results The repository prints no numeric results for any task; 'Create model leaderboard for each task' is an unchecked item on its to-do list (BacBench README 'To-do-list'; the README has no results table)
- Bacformer base parameter count printed in the BacBench README: Parameter count 27M (BacBench README, 'Benchmarked models' table, row 'Bacformer', column 'Params')
- Bacformer Large parameter count printed in the BacBench README: Parameter count 27M (BacBench README, 'Benchmarked models' table, row 'Bacformer Large', columns 'Params' and 'dim')
- Bacformer's authors also built and ran the benchmark: Independence The same group built Bacformer, built BacBench, curated every task dataset and ran all twelve benchmarked models, including the comparators (Preprint 'Author Contributions' and Methods 'Benchmarking'; BacBench README, which is in the same author's namespace)
- Model table differences between the BacBench README and the preprint: Model table differences The BacBench README and preprint Table S9 disagree on Nucleotide Transformer (250M against 235M), ProkBERT (27M against 26M), ESM-C (300M against 332M), ProtBert (420M against 412M), Bacformer Large (27M against 332M) and on the ESM-C context length (2,048 against 1,024 tokens) (BacBench README 'Benchmarked models' table against preprint Table S9)
- Where each BacBench task's printed results are: Result coverage All five active tasks have printed numeric results in the supplementary tables of version 3 of the Bacformer preprint: operon identification in Table S1, protein-protein interaction in Table S13, essential genes in Table S14, phenotypic traits in Table S16 and antibiotic resistance in Table S18; the deprecated strain clustering task is in Table S12 and a protein function prediction task that BacBench does not list is in Table S15 (Preprint Tables S1, S12, S13, S14, S15, S16 and S18)
Models, benchmarks and other records
Models (1)
Configurations (2)
Benchmarks (1)
Tasks (6)
Datasets (6)
- BacBench antibiotic resistance dataset from the NCBI AST browser
- BacBench essential gene dataset from the Database of Essential Genes
- BacBench operon identification dataset from long-read RNA sequencing
- BacBench phenotypic trait dataset
- BacBench protein-protein interaction dataset from STRING DB
- BacBench strain clustering dataset of 6,071 MGnify MAGs
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
27 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Artifact SHA-256 c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e Source metadata | BacBench repository README at commit 64b7d45 No field-specific location recorded Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Artifact url https://raw.githubusercontent.com/macwiatrak/BacBench/64b7d45823b2ff2ef88458e3aa235caa461a12aa/README.md Source metadata | BacBench repository README at commit 64b7d45 No field-specific location recorded Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Code url https://github.com/macwiatrak/BacBench Source metadata | BacBench repository README at commit 64b7d45 No field-specific location recorded Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Limitations3 values
| BacBench repository README at commit 64b7d45 No field-specific location recorded Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Media type text/markdown Source metadata | BacBench repository README at commit 64b7d45 No field-specific location recorded Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Publication status official_project_source Source metadata | BacBench repository README at commit 64b7d45 No field-specific location recorded Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Retrieved at 2026-11-15T23:09:36Z Source metadata | BacBench repository README at commit 64b7d45 No field-specific location recorded Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Review: artifact sha256 c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e Source metadata | BacBench repository README at commit 64b7d45 No field-specific location recorded Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Review: method note Independent review pass. Each source artifact was retrieved again at review time and its SHA-256 compared with the source record; the quoted statement was then read at the recorded locator and compared with the recorded value. Access states were rechecked rather than taken from notes.md. Source metadata | BacBench repository README at commit 64b7d45 No field-specific location recorded Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Review method3 values
| BacBench repository README at commit 64b7d45 No field-specific location recorded Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
Release 2026-11-21-adc15377fab2 · Record review: source checked
The BacBench repository README, which states the active and deprecated tasks, the benchmarked models and the dataset locations.
Technical metadata and extraction receipts
Stable ID: bacformer-20261115-source-bacbench-readme
- areas
- microbes-communities
- contexts
- research
- artifact sha256
- c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e
- artifact url
- https://raw.githubusercontent.com/macwiatrak/BacBench/64b7d45823b2ff2ef88458e3aa235caa461a12aa/README.md
- code url
- https://github.com/macwiatrak/BacBench
- limitations
- Prints no numeric task results; a model leaderboard is on its to-do list.; Its 'Benchmarked models' table gives Bacformer Large 27M parameters, which every other source contradicts, and differs from the preprint on several other parameter counts and context lengths.; Mutable repository file; pinned here to one commit.
- media type
- text/markdown
- publication status
- official_project_source
- retrieved at
- 2026-11-15T23:09:36Z
- review
- artifact sha256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e; method: transcription; source-hash-verification; ai-assisted-source-review; method note: Independent review pass. Each source artifact was retrieved again at review time and its SHA-256 compared with the source record; the quoted statement was then read at the recorded locator and compared with the recorded value. Access states were rechecked rather than taken from notes.md.; note: Statement re-read against the pinned artifact by a reviewer who did not extract this batch. Every artifact except the two preprint PDFs was downloaded again at review time and hashed to exactly the value its source record records, and is byte-identical to the copy kept in artifacts/.; retrieval url: https://raw.githubusercontent.com/macwiatrak/BacBench/64b7d45823b2ff2ef88458e3aa235caa461a12aa/README.md; reviewed at: 2026-11-15T23:55:00Z; reviewer: claude; reviewer note: Claude (Opus 5) review agent, independent of the agent that extracted this batch; no human review claimed. Transcription was checked against the pinned sources. Nothing was reproduced and no value was recomputed.
- source locator
- Sections 'News', 'Datasets', 'Benchmarked models' and 'Task overview'
- url
- https://github.com/macwiatrak/BacBench/blob/64b7d45823b2ff2ef88458e3aa235caa461a12aa/README.md
- version
- Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Related records
- companion of: Bacformer preprint, bioRxiv 2025.07.20.665723 version 3