science-machine/biomlbench repository at commit 673f3d8

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 2 configurations
• 1 benchmark
• 17 tasks
• 3 dataset subsets
• 11 evidence claims

Claims

Statements this source supports, each with where in the source it was checked.

• benchmark-biomlbench: attributes.licence: Licence MIT License (Repository LICENSE at commit 673f3d8 (GitHub licence metadata: MIT))
• benchmark-biomlbench: attributes.limitations: Limitations Task configs set time_limit 28800 s (8 hours) for 18 of the 20 in-scope tasks, and the runner uses it in place of the agent's own TIME_LIMIT_SECS. Q8EG35_SHEON_Campbell_2022_indels and CSN4_MOUSE_Tsuboyama_2023_1UFM_indels set no time_limit, so the agent default applies: 43200 s for AIDE, Biomni and MLAgentBench and 57600 s for STELLA. Imaging tasks set 57600 s. (Repository biomlbench/agents.py (task-specific time limit override), agents/registry.py get_agent_with_task_overrides, biomlbench/tasks/*/config.yaml time_limit, and agents/*/config.yaml time_limit, at commit 673f3d8)
• benchmark-biomlbench: attributes.version: Version 24 task IDs in experiments/biomlbench_v0.1a.txt: 6 proteingym-dms, 5 manual (Open Problems), 9 polarishub and 4 kaggle tasks. (Repository experiments/biomlbench_v0.1a.txt at commit 673f3d8)
• config-aide-gpt-4-1: attributes.parameters: Parameters Repository default for AIDE: agent.code.model and agent.feedback.model gpt-4.1-2025-04-14; installed from git+https://github.com/AIxBio/aideml_bmlb.git (Repository agents/aide/config.yaml and agents/aide/requirements.txt at commit 673f3d8)
• config-biomni-claude-sonnet-4: attributes.parameters: Parameters Repository default for Biomni: llm: claude-sonnet-4-20250514 (Repository agents/biomni/config.yaml at commit 673f3d8, key biomni.kwargs.llm)
• config-mlagentbench-gpt-4-1: attributes.parameters: Parameters Repository default for MLAgentBench: llm-name, edit-script-llm-name and fast-llm-name gpt-4.1-2025-04-14 (Repository agents/mlagentbench/config.yaml at commit 673f3d8)
• config-stella-openrouter-default: attributes.parameters: Parameters Repository default for STELLA: model: claude-sonnet-4, with OpenRouter, Anthropic, Mem0 and Gemini API keys passed; time_limit 57600 seconds (Repository agents/stella/config.yaml at commit 673f3d8)
• data-protein-engineering-tasks: attributes.limitations[0]: Limitations[0] The paper's Table 3 and the repository task list disagree on two protein engineering assays (SPA_STAAU versus SBI_STAAM; PSAE_PICP2 indels versus substitutions); see the source limitations. (Table 3; repository experiments/biomlbench_v0.1a.txt at commit 673f3d8)
• source-miller2025-v3: attributes.limitations[2]: Limitations[2] Table 3 names two protein engineering tasks differently from the repository's task list at commit 673f3d8 (experiments/biomlbench_v0.1a.txt): SPA_STAAU_Tsuboyama_2023 (protein A) versus SBI_STAAM_Tsuboyama_2023_2JVG (immunoglobulin-binding protein Sbi, single substitutions), and PSAE_PICP2_indels versus PSAE_PICP2_Tsuboyama_2023_1PSE (the substitution assay; the repository also holds a separate _indels task). Which of these assays were run is unresolved. The third label difference, CBX4_HUMAN_multi-sub versus CBX4_HUMAN_Tsuboyama_2023_2K28, is the same assay: the repository description of 2K28 says it contains multi-substitution variants of CBX4_HUMAN. (Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8)
• task-cbx4-human-multi-sub: attributes.dataset: Dataset ProteinGym DMS assay CBX4_HUMAN_Tsuboyama_2023_2K28 (repository task list); its description.md says it contains multi-substitution variants of CBX4_HUMAN, matching Table 3 (Repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/CBX4_HUMAN_Tsuboyama_2023_2K28/description.md at commit 673f3d8)
• task-cbx4-human-multi-sub: attributes.scope_note: Scope note Table 3 labels this task CBX4_HUMAN_multi-sub; the repository task list names CBX4_HUMAN_Tsuboyama_2023_2K28, whose description says it contains multi-substitution variants of CBX4_HUMAN. Treated as the same assay under a shortened label; recorded under the paper's label. (Repository biomlbench/tasks/proteingym-dms/CBX4_HUMAN_Tsuboyama_2023_2K28/description.md at commit 673f3d8)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.