BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML
bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) · Preprint
What this source contributed
Records in this release that cite this source. A record can cite several sources.
Results
Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.
Claims
Statements this source supports, each with where in the source it was checked.
- benchmark-biomlbench: attributes.limitations: Limitations Agent scores are ranked against human leaderboards which are largely expert-populated; public leaderboards inevitably reflect a mix of participant expertise, effort, and compute budgets. (Figure 1 caption; Section 5.1, second paragraph)
- benchmark-biomlbench: attributes.limitations: Limitations The framework is adapted from MLE-bench, but with key modifications to support biomedical contexts. (Section 3.1, first paragraph)
- benchmark-biomlbench: attributes.limitations: Limitations Imaging tasks used a single NVIDIA L4 GPU (16 vCPUs, 64 GB RAM) with a 16-hour wall-clock limit. All other tasks used CPU-only machines with identical specifications and an 8-hour limit. (Section 3.4, last paragraph)
- benchmark-biomlbench: attributes.limitations: Limitations At evaluation time, we used the latest compatible LLM backends: GPT-4.1 for MLAgentBench and AIDE, Claude Sonnet 4 for Biomni, and the OpenRouter default for STELLA. (Section 3.3)
- benchmark-biomlbench: attributes.scope_note: Scope note BioML-bench spans four domains (protein engineering, single-cell omics, biomedical imaging, and drug discovery) with tasks that require agents to parse a task description, build a pipeline, implement models, and submit predictions graded by established metrics (e.g., AUROC, Spearman). (Abstract)
- config-aide-gpt-4-1: attributes.version: Version AIDE (v6.3.3) (Section 3.3)
- config-biomni-claude-sonnet-4: attributes.version: Version Biomni (specifically, v0.0.5) (Section 3.3)
- data-protein-engineering-tasks: attributes.limitations[0]: Limitations[0] The paper's Table 3 and the repository task list disagree on two protein engineering assays (SPA_STAAU versus SBI_STAAM; PSAE_PICP2 indels versus substitutions); see the source limitations. (Table 3; repository experiments/biomlbench_v0.1a.txt at commit 673f3d8)
- source-miller2025-v3: attributes.limitations[2]: Limitations[2] Table 3 names two protein engineering tasks differently from the repository's task list at commit 673f3d8 (experiments/biomlbench_v0.1a.txt): SPA_STAAU_Tsuboyama_2023 (protein A) versus SBI_STAAM_Tsuboyama_2023_2JVG (immunoglobulin-binding protein Sbi, single substitutions), and PSAE_PICP2_indels versus PSAE_PICP2_Tsuboyama_2023_1PSE (the substitution assay; the repository also holds a separate _indels task). Which of these assays were run is unresolved. The third label difference, CBX4_HUMAN_multi-sub versus CBX4_HUMAN_Tsuboyama_2023_2K28, is the same assay: the repository description of 2K28 says it contains multi-substitution variants of CBX4_HUMAN. (Table 3, protein engineering rows; repository experiments/biomlbench_v0.1a.txt and biomlbench/tasks/proteingym-dms/{SBI_STAAM_Tsuboyama_2023_2JVG,PSAE_PICP2_Tsuboyama_2023_1PSE,CBX4_HUMAN_Tsuboyama_2023_2K28}/description.md at commit 673f3d8)
Models, benchmarks and other records
Models (2)
Methods (4)
Configurations (5)
Benchmarks (1)
Tasks (20)
- BioML-bench task CBX4_HUMAN_multi-sub
- BioML-bench task CSN4_MOUSE_indels
- BioML-bench task open-problems-cell-communication
- BioML-bench task open-problems-label-projection
- BioML-bench task open-problems-modality
- BioML-bench task open-problems-perturbations
- BioML-bench task open-problems-spatial-genes
- BioML-bench task polaris-adme-fang-hclint-1
- BioML-bench task polaris-adme-fang-hppb-1
- BioML-bench task polaris-adme-fang-solu-1
- BioML-bench task polaris-pkis2-egfr-wt-c-1
- BioML-bench task PSAE_PICP2_indels
- BioML-bench task Q8EG35_SHEON_indels
- BioML-bench task SPA_STAAU_Tsuboyama_2023
- BioML-bench task SPIKE_SARS2_Starr_2020
- BioML-bench task tdcommons-bbb-martins
- BioML-bench task tdcommons-caco2-wang
- BioML-bench task tdcommons-cyp2d6-substrate
- BioML-bench task tdcommons-herg
- BioML-bench task tdcommons-lipophilicity
Protocols (3)
Dataset subsets (3)
Evaluations (30)
- AIDE on BioML-bench drug discovery (non-penalized)
- AIDE on BioML-bench drug discovery (penalized)
- AIDE on BioML-bench protein engineering (non-penalized)
- AIDE on BioML-bench protein engineering (penalized)
- AIDE on BioML-bench single cell omics (non-penalized)
- AIDE on BioML-bench single cell omics (penalized)
- Biomni on BioML-bench drug discovery (non-penalized)
- Biomni on BioML-bench drug discovery (penalized)
- Biomni on BioML-bench protein engineering (non-penalized)
- Biomni on BioML-bench protein engineering (penalized)
- Biomni on BioML-bench single cell omics (non-penalized)
- Biomni on BioML-bench single cell omics (penalized)
- Dummy on BioML-bench drug discovery (non-penalized)
- Dummy on BioML-bench drug discovery (penalized)
- Dummy on BioML-bench protein engineering (non-penalized)
- Dummy on BioML-bench protein engineering (penalized)
- Dummy on BioML-bench single cell omics (non-penalized)
- Dummy on BioML-bench single cell omics (penalized)
- MLAgentBench on BioML-bench drug discovery (non-penalized)
- MLAgentBench on BioML-bench drug discovery (penalized)
- MLAgentBench on BioML-bench protein engineering (non-penalized)
- MLAgentBench on BioML-bench protein engineering (penalized)
- MLAgentBench on BioML-bench single cell omics (non-penalized)
- MLAgentBench on BioML-bench single cell omics (penalized)
- STELLA on BioML-bench drug discovery (non-penalized)
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
23 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Access and reuse CC-BY-4.0 permits redistribution of the article bytes. Source metadata | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML No field-specific location recorded Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Archive note Not archived in the batch: the PDF is 2,282,513 bytes and the repository keeps releases small. The artifact_sha256 pins the bytes that were read. Source metadata | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML No field-specific location recorded Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Artifact SHA-256 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35 Source metadata | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML No field-specific location recorded Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Artifact url https://www.biorxiv.org/content/10.1101/2025.09.01.673319v3.full.pdf Source metadata | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML No field-specific location recorded Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Doi 10.1101/2025.09.01.673319 Source metadata | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML No field-specific location recorded Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Extraction method pdftotext -layout text layer, parsed by extract/extract_biomlbench.py Source metadata | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML No field-specific location recorded Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Hash scope SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. Source metadata | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML No field-specific location recorded Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Licence CC-BY-4.0 Source metadata | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML No field-specific location recorded Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
Limitations10 values
| BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML No field-specific location recorded Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Media type application/pdf Source metadata | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML No field-specific location recorded Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
Sources and history
Release 2026-11-02-a7f03c9e55a5 · Record review: source checked
bioRxiv preprint (version 3) introducing BioML-bench and reporting four agents against human leaderboards.
Technical metadata and extraction receipts
Stable ID: lit-agents-20261013-biomlbench-source-miller2025-v3
- areas
- molecular-interactions; protein-function; single-cell
- contexts
- research
- url
- https://www.biorxiv.org/content/10.1101/2025.09.01.673319v3
- artifact url
- https://www.biorxiv.org/content/10.1101/2025.09.01.673319v3.full.pdf
- version
- bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07)
- retrieved at
- 2026-10-13T12:38:42Z
- artifact sha256
- 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35
- doi
- 10.1101/2025.09.01.673319
- publication status
- preprint
- licence
- CC-BY-4.0
- media type
- application/pdf
- source locator
- Full artifact bytes; per-record locators on each record
- access and reuse
- CC-BY-4.0 permits redistribution of the article bytes.
- archive note
- Not archived in the batch: the PDF is 2,282,513 bytes and the repository keeps releases small. The artifact_sha256 pins the bytes that were read.
- extraction method
- pdftotext -layout text layer, parsed by extract/extract_biomlbench.py
- limitations
- Preprint, not peer reviewed. No published version was found (bioRxiv publication API, 2026-10-13).; Results are printed only as aggregates: Table 2 over all 24 tasks and Table 4 per domain. Per-task leaderboard percentiles appear only as points in Figures 2 and 4, with no printed values.; Table 3 names two protein engineering tasks differently from the repository's task list at commit 673f3d8 (experiments/biomlbench_v0.1a.txt): SPA_STAAU_Tsuboyama_2023 (protein A) versus SBI_STAAM_Tsuboyama_2023_2JVG (immunoglobulin-binding protein Sbi, single substitutions), and PSAE_PICP2_indels versus PSAE_PICP2_Tsuboyama_2023_1PSE (the substitution assay; the repository also holds a separate _indels task). Which of these assays were run is unresolved. The third label difference, CBX4_HUMAN_multi-sub versus CBX4_HUMAN_Tsuboyama_2023_2K28, is the same assay: the repository description of 2K28 says it contains multi-substitution variants of CBX4_HUMAN.; Table 4 prints a value with a plus-or-minus spread but its caption does not say what the spread is. Figures 2 and 4 show the same per-domain leaderboard percentiles as mean ± SEM across tasks, and the Table 2 caption refers to the reported SEM.; Section 3.4 defines Above Median, Any Medal and Completion Rate over tasks, but the Table 4 values are multiples of 1/(4 x tasks in the domain) (for example 45.8% = 11/24 for 6 protein tasks), which fits counting replicate runs. The denominator is unresolved.; Table 4 prints Completion Rate 'NA' for Dummy in the penalized block, while Table 2 prints 100.0 for Dummy overall.; All agents were run by the benchmark's authors, who did not build them. B. T. is CTO of ScienceMachine, which provided compute and LLM API resources (Competing Interests; Acknowledgments).; Version 3 replaced the STELLA rows for Drug Discovery and Protein Engineering in Table 4 (and STELLA's Table 2 rows) with new values without explaining why in the text; for example STELLA's penalized protein engineering leaderboard percentile is 14.78 in v2 and 34.98 in v3, and its completion rate 87.5 and 100.0. The other agents' rows are unchanged, including their Mean Rank, which leaves the Mean Rank columns of those blocks internally inconsistent (they no longer sum to 15). Those 20 Mean Rank results are marked disputed.; Section 4.2 says Biomni leads Protein Engineering with penalties applied; that sentence is unchanged from v2. In v3 Table 4 STELLA has the higher penalized protein engineering leaderboard percentile (34.98 vs 29.93) and Above Median (45.8 vs 25.0); Biomni has the better Mean Rank (2.00 vs 2.33).; M. G. is a contract employee of ScienceMachine, as well as B. T. being its CTO (Competing Interests).
- hash scope
- SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer.