rewirebio.iobenchmarks
Source

BacBench repository README at commit 64b7d45

Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa · Project source

Original source

What this source contributed

Records in this release that cite this source. A record can cite several sources.

Claims

Statements this source supports, each with where in the source it was checked.

  • Apache-2.0 covers the BacBench code: Licence Apache License 2.0, stated by the README licence badge and carried by the LICENSE file at the pinned commit (BacBench README licence badge, line 3; LICENSE file at the pinned commit)
  • BacBench changed the essential gene and PPI splits on 2026-05-13: Protocol change On 13 May 2026 BacBench updated the essential gene and protein-protein interaction tasks to a phylogeny-aware split by genus, added BacLM as a benchmarked model and deprecated the strain clustering task (BacBench README 'News', entry dated 13.05.2026)
  • BacBench has five active tasks and one deprecated task: Tasks Five active tasks (essential gene prediction, operon identification from long-read RNA sequencing, protein-protein interaction, antibiotic resistance prediction and phenotypic trait prediction) and one deprecated task (strain clustering), which remains in the repository for reproducibility (BacBench README opening paragraph and 'Task overview' table)
  • BacBench publishes no numeric results: Reported results The repository prints no numeric results for any task; 'Create model leaderboard for each task' is an unchecked item on its to-do list (BacBench README 'To-do-list'; the README has no results table)
  • Bacformer base parameter count printed in the BacBench README: Parameter count 27M (BacBench README, 'Benchmarked models' table, row 'Bacformer', column 'Params')
  • Bacformer Large parameter count printed in the BacBench README: Parameter count 27M (BacBench README, 'Benchmarked models' table, row 'Bacformer Large', columns 'Params' and 'dim')
  • Bacformer's authors also built and ran the benchmark: Independence The same group built Bacformer, built BacBench, curated every task dataset and ran all twelve benchmarked models, including the comparators (Preprint 'Author Contributions' and Methods 'Benchmarking'; BacBench README, which is in the same author's namespace)
  • Model table differences between the BacBench README and the preprint: Model table differences The BacBench README and preprint Table S9 disagree on Nucleotide Transformer (250M against 235M), ProkBERT (27M against 26M), ESM-C (300M against 332M), ProtBert (420M against 412M), Bacformer Large (27M against 332M) and on the ESM-C context length (2,048 against 1,024 tokens) (BacBench README 'Benchmarked models' table against preprint Table S9)
  • Where each BacBench task's printed results are: Result coverage All five active tasks have printed numeric results in the supplementary tables of version 3 of the Bacformer preprint: operon identification in Table S1, protein-protein interaction in Table S13, essential genes in Table S14, phenotypic traits in Table S16 and antibiotic resistance in Table S18; the deprecated strain clustering task is in Table S12 and a protein function prediction task that BacBench does not list is in Table S15 (Preprint Tables S1, S12, S13, S14, S15, S16 and S18)

Models, benchmarks and other records

Models (1)

Configurations (2)

Benchmarks (1)

Tasks (6)

Datasets (6)

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

27 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-21-adc15377fab2
Property and statementOriginal source and locationReview and provenance
Artifact SHA-256
c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e
Source metadata
BacBench repository README at commit 64b7d45

Original source ↗

No field-specific location recorded

Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Retrieved: 2026-11-15T23:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.artifact_sha256

Source artifact SHA-256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Artifact url
https://raw.githubusercontent.com/macwiatrak/BacBench/64b7d45823b2ff2ef88458e3aa235caa461a12aa/README.md
Source metadata
BacBench repository README at commit 64b7d45

Original source ↗

No field-specific location recorded

Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Retrieved: 2026-11-15T23:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.artifact_url

Source artifact SHA-256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Code url
https://github.com/macwiatrak/BacBench
Source metadata
BacBench repository README at commit 64b7d45

Original source ↗

No field-specific location recorded

Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Retrieved: 2026-11-15T23:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.code_url

Source artifact SHA-256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Limitations
3 values
  • Prints no numeric task results; a model leaderboard is on its to-do list.
  • Its 'Benchmarked models' table gives Bacformer Large 27M parameters, which every other source contradicts, and differs from the preprint on several other parameter counts and context lengths.
  • Mutable repository file; pinned here to one commit.
Source metadata
BacBench repository README at commit 64b7d45

Original source ↗

No field-specific location recorded

Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Retrieved: 2026-11-15T23:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.limitations

Source artifact SHA-256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Media type
text/markdown
Source metadata
BacBench repository README at commit 64b7d45

Original source ↗

No field-specific location recorded

Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Retrieved: 2026-11-15T23:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.media_type

Source artifact SHA-256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Publication status
official_project_source
Source metadata
BacBench repository README at commit 64b7d45

Original source ↗

No field-specific location recorded

Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Retrieved: 2026-11-15T23:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.publication_status

Source artifact SHA-256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Retrieved at
2026-11-15T23:09:36Z
Source metadata
BacBench repository README at commit 64b7d45

Original source ↗

No field-specific location recorded

Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Retrieved: 2026-11-15T23:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.retrieved_at

Source artifact SHA-256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Review: artifact sha256
c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e
Source metadata
BacBench repository README at commit 64b7d45

Original source ↗

No field-specific location recorded

Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Retrieved: 2026-11-15T23:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.review.artifact_sha256

Source artifact SHA-256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Review: method note
Independent review pass. Each source artifact was retrieved again at review time and its SHA-256 compared with the source record; the quoted statement was then read at the recorded locator and compared with the recorded value. Access states were rechecked rather than taken from notes.md.
Source metadata
BacBench repository README at commit 64b7d45

Original source ↗

No field-specific location recorded

Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Retrieved: 2026-11-15T23:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.review.method_note

Source artifact SHA-256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Review method
3 values
  • transcription
  • source-hash-verification
  • ai-assisted-source-review
Source metadata
BacBench repository README at commit 64b7d45

Original source ↗

No field-specific location recorded

Version: Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Retrieved: 2026-11-15T23:09:36Z

catalogued

No individual claim review recorded

Audit details

Field: attributes.review.method

Source artifact SHA-256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-21-adc15377fab2 · Record review: source checked

The BacBench repository README, which states the active and deprecated tasks, the benchmarked models and the dataset locations.

Release history

No supporting source is linked yet.

Read original source

Download this release (gzip)
Technical metadata and extraction receipts

Stable ID: bacformer-20261115-source-bacbench-readme

areas
microbes-communities
contexts
research
artifact sha256
c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e
artifact url
https://raw.githubusercontent.com/macwiatrak/BacBench/64b7d45823b2ff2ef88458e3aa235caa461a12aa/README.md
code url
https://github.com/macwiatrak/BacBench
limitations
Prints no numeric task results; a model leaderboard is on its to-do list.; Its 'Benchmarked models' table gives Bacformer Large 27M parameters, which every other source contradicts, and differs from the preprint on several other parameter counts and context lengths.; Mutable repository file; pinned here to one commit.
media type
text/markdown
publication status
official_project_source
retrieved at
2026-11-15T23:09:36Z
review
artifact sha256: c9d28efef5b4cc30adbcc9de224c350100f5aa2a15a025beaf10dcdeb86cd93e; method: transcription; source-hash-verification; ai-assisted-source-review; method note: Independent review pass. Each source artifact was retrieved again at review time and its SHA-256 compared with the source record; the quoted statement was then read at the recorded locator and compared with the recorded value. Access states were rechecked rather than taken from notes.md.; note: Statement re-read against the pinned artifact by a reviewer who did not extract this batch. Every artifact except the two preprint PDFs was downloaded again at review time and hashed to exactly the value its source record records, and is byte-identical to the copy kept in artifacts/.; retrieval url: https://raw.githubusercontent.com/macwiatrak/BacBench/64b7d45823b2ff2ef88458e3aa235caa461a12aa/README.md; reviewed at: 2026-11-15T23:55:00Z; reviewer: claude; reviewer note: Claude (Opus 5) review agent, independent of the agent that extracted this batch; no human review claimed. Transcription was checked against the pinned sources. Nothing was reproduced and no value was recomputed.
source locator
Sections 'News', 'Datasets', 'Benchmarked models' and 'Task overview'
url
https://github.com/macwiatrak/BacBench/blob/64b7d45823b2ff2ef88458e3aa235caa461a12aa/README.md
version
Commit 64b7d45823b2ff2ef88458e3aa235caa461a12aa
Related records

Suggest a correction