rewirebio.iobenchmarks
← Browse all benchmarks
Benchmark

BioML-bench

Benchmark of autonomous LLM agents on end-to-end biomedical machine learning: an agent reads a task description, builds and runs its own pipeline and submits test-set predictions, which are graded by the task's original metric and placed on that task's public human leaderboard.

Results are recorded on the linked protocols, tasks or configurations listed below. Their settings and evaluations are kept separate.

3 protocols with results

Overview

An explanatory profile has not yet been reviewed for this record.

Consult the linked sources for architecture or protocol details. Missing evidence is not evidence of a missing capability.

Results on linked records

This benchmark has no results of its own in this release. 3 linked records have results. They are not assigned to the underlying benchmark: each record keeps its own settings and evaluations, and its scores are on its own page.

Methods and evaluation design

Procedure, tasks and evaluated configurations

Additional methodological details, tasks and evaluated configurations have not yet been extracted into a structured profile for this benchmark. Consult the linked sources and evaluation records below.

Baseline coverage

Reference methods help show what a model adds beyond simple controls. We track a null control and a conventional method for each protocol.

0 of 6 active baseline roles have published Rewire measurements in this release. Measurements on a selected protocol do not establish coverage of an entire suite.

Baseline status by linked protocol

Protocol coverage CSV (gzip) · Model evaluation matrix (gzip) · Source table (gzip) · Release and checksums (gzip)

Coverage is derived from release 2026-11-02-a7f03c9e55a5. Source citations describe the original records; they do not validate an unreviewed baseline proposal. No results have been generated by this audit.

Run instructions

No runnable recipe has been reviewed for this benchmark. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

7 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-02-a7f03c9e55a5
Property and statementOriginal source and locationReview and provenance
Limitations
3 values
  • Scores are leaderboard percentiles: they depend on who submitted to each public leaderboard and when, not only on the task metric.
  • Results are per-domain aggregates over 5 to 9 tasks; per-task values are not printed.
  • Built on the MLE-bench scaffold; tasks are repackaged from Polaris Hub, ProteinGym and Open Problems with BioML-bench's own preparation scripts.
Individual claims		BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML

Original source ↗

Figure 1 caption; Section 5.1, second paragraph

Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07)
Retrieved: 2026-10-13T12:38:42Z

conflicting claim

["source-hash-verification","pdf-text-parse","independent-cell-check"] · 2026-10-13

Audit details

Independent review 2026-10-13: wording found at the locator.

Conflicting source claim: "Agent scores are ranked against human leaderboards which are largely expert-populated; public leaderboards inevitably reflect a mix of participant expertise, effort, and compute budgets.". Claim record status: source_checked.

Field: attributes.limitations

Claim: lit-agents-20261013-biomlbench-claim-benchmark-human-baselines

Source artifact SHA-256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35

Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer.

Inspected artifact

Extraction artifact SHA-256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35

Extraction artifact

Limitations
3 values
  • Scores are leaderboard percentiles: they depend on who submitted to each public leaderboard and when, not only on the task metric.
  • Results are per-domain aggregates over 5 to 9 tasks; per-task values are not printed.
  • Built on the MLE-bench scaffold; tasks are repackaged from Polaris Hub, ProteinGym and Open Problems with BioML-bench's own preparation scripts.
Individual claims		BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML

Original source ↗

Section 3.1, first paragraph

Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07)
Retrieved: 2026-10-13T12:38:42Z

conflicting claim

["source-hash-verification","pdf-text-parse","independent-cell-check"] · 2026-10-13

Audit details

Independent review 2026-10-13: wording found at the locator.

Conflicting source claim: "The framework is adapted from MLE-bench, but with key modifications to support biomedical contexts.". Claim record status: source_checked.

Field: attributes.limitations

Claim: lit-agents-20261013-biomlbench-claim-benchmark-mle-bench

Source artifact SHA-256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35

Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer.

Inspected artifact

Extraction artifact SHA-256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35

Extraction artifact

Limitations
3 values
  • Scores are leaderboard percentiles: they depend on who submitted to each public leaderboard and when, not only on the task metric.
  • Results are per-domain aggregates over 5 to 9 tasks; per-task values are not printed.
  • Built on the MLE-bench scaffold; tasks are repackaged from Polaris Hub, ProteinGym and Open Problems with BioML-bench's own preparation scripts.
Individual claims		BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML

Original source ↗

Section 3.4, last paragraph

Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07)
Retrieved: 2026-10-13T12:38:42Z

conflicting claim

["source-hash-verification","pdf-text-parse","independent-cell-check"] · 2026-10-13

Audit details

Independent review 2026-10-13: wording found at the locator.

Conflicting source claim: "Imaging tasks used a single NVIDIA L4 GPU (16 vCPUs, 64 GB RAM) with a 16-hour wall-clock limit. All other tasks used CPU-only machines with identical specifications and an 8-hour limit.". Claim record status: source_checked.

Field: attributes.limitations

Claim: lit-agents-20261013-biomlbench-claim-compute-budget

Source artifact SHA-256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35

Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer.

Inspected artifact

Extraction artifact SHA-256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35

Extraction artifact

Limitations
3 values
  • Scores are leaderboard percentiles: they depend on who submitted to each public leaderboard and when, not only on the task metric.
  • Results are per-domain aggregates over 5 to 9 tasks; per-task values are not printed.
  • Built on the MLE-bench scaffold; tasks are repackaged from Polaris Hub, ProteinGym and Open Problems with BioML-bench's own preparation scripts.
Individual claims		BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML

Original source ↗

Section 3.3

Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07)
Retrieved: 2026-10-13T12:38:42Z

conflicting claim

["source-hash-verification","pdf-text-parse","independent-cell-check"] · 2026-10-13

Audit details

Independent review 2026-10-13: wording found at the locator.

Conflicting source claim: "At evaluation time, we used the latest compatible LLM backends: GPT-4.1 for MLAgentBench and AIDE, Claude Sonnet 4 for Biomni, and the OpenRouter default for STELLA.". Claim record status: source_checked.

Field: attributes.limitations

Claim: lit-agents-20261013-biomlbench-claim-llm-backends

Source artifact SHA-256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35

Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer.

Inspected artifact

Extraction artifact SHA-256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35

Extraction artifact

Limitations
3 values
  • Scores are leaderboard percentiles: they depend on who submitted to each public leaderboard and when, not only on the task metric.
  • Results are per-domain aggregates over 5 to 9 tasks; per-task values are not printed.
  • Built on the MLE-bench scaffold; tasks are repackaged from Polaris Hub, ProteinGym and Open Problems with BioML-bench's own preparation scripts.
Individual claims		science-machine/biomlbench repository at commit 673f3d8

Original source ↗

Repository biomlbench/agents.py (task-specific time limit override), agents/registry.py get_agent_with_task_overrides, biomlbench/tasks/*/config.yaml time_limit, and agents/*/config.yaml time_limit, at commit 673f3d8

Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)'
Retrieved: 2026-10-13T12:38:10Z

conflicting claim

["source-hash-verification","independent-cell-check"] · 2026-10-13

Audit details

Added by the independent review 2026-10-13: the extractor's notes read the agent defaults as the run limits; the per-task limit overrides them for 18 of 20 in-scope tasks.

Conflicting source claim: "Task configs set time_limit 28800 s (8 hours) for 18 of the 20 in-scope tasks, and the runner uses it in place of the agent's own TIME_LIMIT_SECS. Q8EG35_SHEON_Campbell_2022_indels and CSN4_MOUSE_Tsuboyama_2023_1UFM_indels set no time_limit, so the agent default applies: 43200 s for AIDE, Biomni and MLAgentBench and 57600 s for STELLA. Imaging tasks set 57600 s.". Claim record status: source_checked.

Field: attributes.limitations

Claim: lit-agents-20261013-biomlbench-claim-repo-time-limits

Source artifact SHA-256: c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559

Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md.

Inspected artifact

Extraction artifact SHA-256: c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559

Extraction artifact

Scope note
Four domains: drug discovery (9 tasks), protein engineering (6), single-cell omics (5) and biomedical imaging (4). Only the three molecular and omics domains are recorded; the imaging domain and the all-task Table 2 aggregate (which includes imaging) are out of scope.
Individual claims		BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML

Original source ↗

Abstract

Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07)
Retrieved: 2026-10-13T12:38:42Z

conflicting claim

["source-hash-verification","pdf-text-parse","independent-cell-check"] · 2026-10-13

Audit details

Independent review 2026-10-13: wording found at the locator.

Conflicting source claim: "BioML-bench spans four domains (protein engineering, single-cell omics, biomedical imaging, and drug discovery) with tasks that require agents to parse a task description, build a pipeline, implement models, and submit predictions graded by established metrics (e.g., AUROC, Spearman).". Claim record status: source_checked.

Field: attributes.scope_note

Claim: lit-agents-20261013-biomlbench-claim-benchmark-domains

Source artifact SHA-256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35

Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer.

Inspected artifact

Extraction artifact SHA-256: 875cb3519dac9b0ec51c35224e1fc90b107b0676a4cc7c22cecb56eb8ac38b35

Extraction artifact

Version
v0.1-alpha; 24-task set listed in experiments/biomlbench_v0.1a.txt at commit 673f3d8 and Table 3 of bioRxiv v3
Individual claims		science-machine/biomlbench repository at commit 673f3d8

Original source ↗

Repository experiments/biomlbench_v0.1a.txt at commit 673f3d8

Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)'
Retrieved: 2026-10-13T12:38:10Z

conflicting claim

["source-hash-verification","independent-cell-check"] · 2026-10-13

Audit details

Independent review 2026-10-13: matched the cited repository file at commit 673f3d8.

Conflicting source claim: "24 task IDs in experiments/biomlbench_v0.1a.txt: 6 proteingym-dms, 5 manual (Open Problems), 9 polarishub and 4 kaggle tasks.". Claim record status: source_checked.

Field: attributes.version

Claim: lit-agents-20261013-biomlbench-claim-benchmark-task-list

Source artifact SHA-256: c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559

Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md.

Inspected artifact

Extraction artifact SHA-256: c23978275652d61452e710dfdf94613237b85ab9b90605af457d082c351ae559

Extraction artifact

How to read the evidence

Sources and history

Release 2026-11-02-a7f03c9e55a5 · Record review: source checked

2 source records and release historyDownload this release (gzip)
Technical metadata and extraction receipts

Stable ID: lit-agents-20261013-biomlbench-benchmark-biomlbench

areas
molecular-interactions; protein-function; single-cell
contexts
research
entity level
suite
version
v0.1-alpha; 24-task set listed in experiments/biomlbench_v0.1a.txt at commit 673f3d8 and Table 3 of bioRxiv v3
task
End-to-end biomedical ML by autonomous agents, scored against public human leaderboards
scope note
Four domains: drug discovery (9 tasks), protein engineering (6), single-cell omics (5) and biomedical imaging (4). Only the three molecular and omics domains are recorded; the imaging domain and the all-task Table 2 aggregate (which includes imaging) are out of scope.
source locator
Sections 3.1-3.4; Table 1; Table 3; repository README
limitations
Scores are leaderboard percentiles: they depend on who submitted to each public leaderboard and when, not only on the task metric.; Results are per-domain aggregates over 5 to 9 tasks; per-task values are not printed.; Built on the MLE-bench scaffold; tasks are repackaged from Polaris Hub, ProteinGym and Open Problems with BioML-bench's own preparation scripts.
Related records

Suggest a correction