source-miller2025-v3: attributes.limitations[2]
Limitations[2] of BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML
Table 3 names two protein engineering tasks differently from the repository's task list at commit 673f3d8 (experiments/biomlbench_v0.1a.txt): SPA_STAAU_Tsuboyama_2023 (protein A) versus SBI_STAAM_Tsuboyama_2023_2JVG (immunoglobulin-binding protein Sbi, single substitutions), and PSAE_PICP2_indels versus PSAE_PICP2_Tsuboyama_2023_1PSE (the substitution assay; the repository also holds a separate _indels task). Which of these assays were run is unresolved. The third label difference, CBX4_HUMAN_multi-sub versus CBX4_HUMAN_Tsuboyama_2023_2K28, is the same assay: the repository description of 2K28 says it contains multi-substitution variants of CBX4_HUMAN.
Two of the three Table 3 label differences are real assay conflicts; the CBX4 one is a shortened label for the same assay.
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
14 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Claimed field attributes.limitations[2] Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Claimed field attributes.limitations[2] Context-only references | science-machine/biomlbench repository at commit 673f3d8 Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. |
| Previous value Table 3 names three protein engineering tasks differently from the repository's task list at commit 673f3d8 (experiments/biomlbench_v0.1a.txt): SPA_STAAU_Tsuboyama_2023 versus SBI_STAAM_Tsuboyama_2023_2JVG, PSAE_PICP2_indels versus PSAE_PICP2_Tsuboyama_2023_1PSE, and CBX4_HUMAN_multi-sub versus CBX4_HUMAN_Tsuboyama_2023_2K28. Which assays were run is unresolved. Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Previous value Table 3 names three protein engineering tasks differently from the repository's task list at commit 673f3d8 (experiments/biomlbench_v0.1a.txt): SPA_STAAU_Tsuboyama_2023 versus SBI_STAAM_Tsuboyama_2023_2JVG, PSAE_PICP2_indels versus PSAE_PICP2_Tsuboyama_2023_1PSE, and CBX4_HUMAN_multi-sub versus CBX4_HUMAN_Tsuboyama_2023_2K28. Which assays were run is unresolved. Context-only references | science-machine/biomlbench repository at commit 673f3d8 Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. |
| Source location Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Source location Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Context-only references | science-machine/biomlbench repository at commit 673f3d8 Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. |
| Claimed value Table 3 names two protein engineering tasks differently from the repository's task list at commit 673f3d8 (experiments/biomlbench_v0.1a.txt): SPA_STAAU_Tsuboyama_2023 (protein A) versus SBI_STAAM_Tsuboyama_2023_2JVG (immunoglobulin-binding protein Sbi, single substitutions), and PSAE_PICP2_indels versus PSAE_PICP2_Tsuboyama_2023_1PSE (the substitution assay; the repository also holds a separate _indels task). Which of these assays were run is unresolved. The third label difference, CBX4_HUMAN_multi-sub versus CBX4_HUMAN_Tsuboyama_2023_2K28, is the same assay: the repository description of 2K28 says it contains multi-substitution variants of CBX4_HUMAN. Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Claimed value Table 3 names two protein engineering tasks differently from the repository's task list at commit 673f3d8 (experiments/biomlbench_v0.1a.txt): SPA_STAAU_Tsuboyama_2023 (protein A) versus SBI_STAAM_Tsuboyama_2023_2JVG (immunoglobulin-binding protein Sbi, single substitutions), and PSAE_PICP2_indels versus PSAE_PICP2_Tsuboyama_2023_1PSE (the substitution assay; the repository also holds a separate _indels task). Which of these assays were run is unresolved. The third label difference, CBX4_HUMAN_multi-sub versus CBX4_HUMAN_Tsuboyama_2023_2K28, is the same assay: the repository description of 2K28 says it contains multi-substitution variants of CBX4_HUMAN. Context-only references | science-machine/biomlbench repository at commit 673f3d8 Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. |
| Description Two of the three Table 3 label differences are real assay conflicts; the CBX4 one is a shortened label for the same assay. Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Description Two of the three Table 3 label differences are real assay conflicts; the CBX4 one is a shortened label for the same assay. Context-only references | science-machine/biomlbench repository at commit 673f3d8 Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. |
Sources and history
Release 2026-11-02-a7f03c9e55a5 · Record review: source checked
2 source records and release history
- BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Original source · bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07)
- science-machine/biomlbench repository at commit 673f3d8 · Original source · Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)'
Technical metadata and extraction receipts
Stable ID: metadata-correction-biomlbench-20261013-source-miller2025-v3-limitations-2
- field
- attributes.limitations[2]
- previous value
- Table 3 names three protein engineering tasks differently from the repository's task list at commit 673f3d8 (experiments/biomlbench_v0.1a.txt): SPA_STAAU_Tsuboyama_2023 versus SBI_STAAM_Tsuboyama_2023_2JVG, PSAE_PICP2_indels versus PSAE_PICP2_Tsuboyama_2023_1PSE, and CBX4_HUMAN_multi-sub versus CBX4_HUMAN_Tsuboyama_2023_2K28. Which assays were run is unresolved.
- value
- Table 3 names two protein engineering tasks differently from the repository's task list at commit 673f3d8 (experiments/biomlbench_v0.1a.txt): SPA_STAAU_Tsuboyama_2023 (protein A) versus SBI_STAAM_Tsuboyama_2023_2JVG (immunoglobulin-binding protein Sbi, single substitutions), and PSAE_PICP2_indels versus PSAE_PICP2_Tsuboyama_2023_1PSE (the substitution assay; the repository also holds a separate _indels task). Which of these assays were run is unresolved. The third label difference, CBX4_HUMAN_multi-sub versus CBX4_HUMAN_Tsuboyama_2023_2K28, is the same assay: the repository description of 2K28 says it contains multi-substitution variants of CBX4_HUMAN.
- source locator
- Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8
- review
- method: source-hash-verification; independent-cell-check; method note: Compared Table 3 of the pinned v3 PDF with experiments/biomlbench_v0.1a.txt and the description.md of each named task in the repository archive at commit 673f3d8.; reviewer: claude; reviewer note: Claude (Opus 5.5) research agent, independent of the extractor; no human review claimed; date: 2026-10-13; artifact sha256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35; retrieval url: https://www.biorxiv.org/content/10.1101/2025.09.01.673319v3.full.pdf; note: Two of the three Table 3 label differences are real assay conflicts; the CBX4 one is a shortened label for the same assay. See docs/reviews/2026-10-13-biomlbench-agents.md.