Evidence claim

benchmark-biomlbench: limitations

Task configs set time_limit 28800 s (8 hours) for 18 of the 20 in-scope tasks, and the runner uses it in place of the agent's own TIME_LIMIT_SECS. Q8EG35_SHEON_Campbell_2022_indels and CSN4_MOUSE_Tsuboyama_2023_1UFM_indels set no time_limit, so the agent default applies: 43200 s for AIDE, Biomni and MLAgentBench and 57600 s for STELLA. Imaging tasks set 57600 s.

Descriptive fact transcribed from the pinned source.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

6 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-02-a7f03c9e55a5
Property and statement	Original source and location	Review and provenance
Claimed field attributes.limitations Context-only references	science-machine/biomlbench repository at commit 673f3d8 Original source ↗ Repository biomlbench/agents.py (task-specific time limit override), agents/registry.py get_agent_with_task_overrides, biomlbench/tasks//config.yaml time_limit, and agents//config.yaml time_limit, at commit 673f3d8 Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' Retrieved: 2026-10-13T12:38:10Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.field` Source artifact SHA-256: `c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559` Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. Inspected artifact
Source location Repository biomlbench/agents.py (task-specific time limit override), agents/registry.py get_agent_with_task_overrides, biomlbench/tasks//config.yaml time_limit, and agents//config.yaml time_limit, at commit 673f3d8 Context-only references	science-machine/biomlbench repository at commit 673f3d8 Original source ↗ Repository biomlbench/agents.py (task-specific time limit override), agents/registry.py get_agent_with_task_overrides, biomlbench/tasks//config.yaml time_limit, and agents//config.yaml time_limit, at commit 673f3d8 Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' Retrieved: 2026-10-13T12:38:10Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.source_locator` Source artifact SHA-256: `c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559` Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. Inspected artifact
Claimed value Task configs set time_limit 28800 s (8 hours) for 18 of the 20 in-scope tasks, and the runner uses it in place of the agent's own TIME_LIMIT_SECS. Q8EG35_SHEON_Campbell_2022_indels and CSN4_MOUSE_Tsuboyama_2023_1UFM_indels set no time_limit, so the agent default applies: 43200 s for AIDE, Biomni and MLAgentBench and 57600 s for STELLA. Imaging tasks set 57600 s. Context-only references	science-machine/biomlbench repository at commit 673f3d8 Original source ↗ Repository biomlbench/agents.py (task-specific time limit override), agents/registry.py get_agent_with_task_overrides, biomlbench/tasks//config.yaml time_limit, and agents//config.yaml time_limit, at commit 673f3d8 Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' Retrieved: 2026-10-13T12:38:10Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.value` Source artifact SHA-256: `c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559` Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. Inspected artifact
Description Descriptive fact transcribed from the pinned source. Context-only references	science-machine/biomlbench repository at commit 673f3d8 Original source ↗ Repository biomlbench/agents.py (task-specific time limit override), agents/registry.py get_agent_with_task_overrides, biomlbench/tasks//config.yaml time_limit, and agents//config.yaml time_limit, at commit 673f3d8 Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' Retrieved: 2026-10-13T12:38:10Z	not individually reviewed No individual claim review recorded Audit details Field: `description` Source artifact SHA-256: `c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559` Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. Inspected artifact
Relationship: subject lit-agents-20261013-biomlbench-benchmark-biomlbench Context-only references	science-machine/biomlbench repository at commit 673f3d8 Original source ↗ Repository biomlbench/agents.py (task-specific time limit override), agents/registry.py get_agent_with_task_overrides, biomlbench/tasks//config.yaml time_limit, and agents//config.yaml time_limit, at commit 673f3d8 Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' Retrieved: 2026-10-13T12:38:10Z	not individually reviewed No individual claim review recorded Audit details Field: `links:subject:lit-agents-20261013-biomlbench-benchmark-biomlbench` Source artifact SHA-256: `c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559` Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. Inspected artifact
Name benchmark-biomlbench: attributes.limitations Context-only references	science-machine/biomlbench repository at commit 673f3d8 Original source ↗ Repository biomlbench/agents.py (task-specific time limit override), agents/registry.py get_agent_with_task_overrides, biomlbench/tasks//config.yaml time_limit, and agents//config.yaml time_limit, at commit 673f3d8 Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' Retrieved: 2026-10-13T12:38:10Z	not individually reviewed No individual claim review recorded Audit details Field: `name` Source artifact SHA-256: `c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559` Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-02-a7f03c9e55a5 · Record review: source checked

1 source record and release history

science-machine/biomlbench repository at commit 673f3d8 · Original source · Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)'

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: lit-agents-20261013-biomlbench-claim-repo-time-limits

areas: molecular-interactions; protein-function; single-cell
contexts: research
field: attributes.limitations
value: Task configs set time_limit 28800 s (8 hours) for 18 of the 20 in-scope tasks, and the runner uses it in place of the agent's own TIME_LIMIT_SECS. Q8EG35_SHEON_Campbell_2022_indels and CSN4_MOUSE_Tsuboyama_2023_1UFM_indels set no time_limit, so the agent default applies: 43200 s for AIDE, Biomni and MLAgentBench and 57600 s for STELLA. Imaging tasks set 57600 s.
source locator: Repository biomlbench/agents.py (task-specific time limit override), agents/registry.py get_agent_with_task_overrides, biomlbench/tasks/*/config.yaml time_limit, and agents/*/config.yaml time_limit, at commit 673f3d8
review: method: source-hash-verification; independent-cell-check; method note: Read the time_limit of each in-scope task config and the override code in the repository archive at commit 673f3d8 (archive SHA-256 matches the source record).; reviewer: claude; reviewer note: Claude (Opus 5.5) research agent, independent of the extractor; no human review claimed; date: 2026-10-13; artifact sha256: c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559; retrieval url: https://codeload.github.com/science-machine/biomlbench/tar.gz/673f3d8d3462fc6cdcbbd9695fba612b9332ca65; note: Added by the independent review 2026-10-13: the extractor's notes read the agent defaults as the run limits; the per-task limit overrides them for 18 of 20 in-scope tasks.

Related records

subject: BioML-bench

Suggest a correction