Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench)
Biomni v0.0.5, Claude Sonnet 4 backend: Biomni (specifically, v0.0.5) with Claude Sonnet 4 as LLM backend; agent class A1 in the repository wrapper (agents/biomni/main.py).
Overview
An explanatory profile has not yet been reviewed for this record.
Consult the linked sources for architecture or protocol details. Missing evidence is not evidence of a missing capability.
Evaluations and results
6 evaluations · 23 results. Different protocols are not a single leaderboard.
Sorted by Leaderboard percentile (mean over tasks in the domain; non-penalized: runs without a valid submission omitted) (higher is better). The best value in each column is highlighted. Decimals are rounded for display; each value links to the printed value and its source.
| Tested configuration | Protocol and dataset | Leaderboard percentile (mean over tasks in the domain; non-penalized: runs without a valid submission omitted) | Leaderboard percentile (mean over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) | Proportion (share of tasks that yield a valid, gradable submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) | Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); non-penalized: runs without a valid submission omitted) | Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); penalized: runs without a valid submission scored as percentile 0) | Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); non-penalized: runs without a valid submission omitted) | Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) | Rank (mean relative rank among the five Table 4 agents (Dummy, MLAgentBench, AIDE, STELLA, Biomni) by leaderboard percentile, over tasks in the domain; non-penalized: runs without a valid submission omitted) | Rank (mean relative rank among the five Table 4 agents (Dummy, MLAgentBench, AIDE, STELLA, Biomni) by leaderboard percentile, over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) |
|---|---|---|---|---|---|---|---|---|---|---|
| Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench)Evaluation | BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks, BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 55.62 ± 10.32% (best in column) | Not reported | Not reported | 30.0 ± 14.6% (best in column) | Not reported | 60.0 ± 20.3% (best in column) | Not reported | 2.20 ± 0.58 | Not reported |
| Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench)Evaluation | BioML-bench drug discovery domain: agent leaderboard percentile over 9 tasks, BioML-bench v0.1-alpha drug discovery tasks (9 tasks) | 32.55 ± 8.61% | Not reported | Not reported | 13.9 ± 11.1% | Not reported | 30.6 ± 12.3% | Not reported | Not reported | Not reported |
| Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench)Evaluation | BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks, BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 29.93 ± 11.04% | Not reported | Not reported | 25.0 ± 17.1% | Not reported | 25.0 ± 17.1% | Not reported | Not reported | Not reported |
| Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench)Evaluation | BioML-bench drug discovery domain: agent leaderboard percentile over 9 tasks, BioML-bench v0.1-alpha drug discovery tasks (9 tasks) | Not reported | 32.55 ± 8.61% | 100% (best in column) | Not reported | 13.9 ± 11.1% | Not reported | 30.6 ± 12.3% | Not reported | Not reported |
| Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench)Evaluation | BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks, BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | Not reported | 29.93 ± 11.04% | 100% (best in column) | Not reported | 25.0 ± 17.1% | Not reported | 25.0 ± 17.1% | Not reported | Not reported |
| Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench)Evaluation | BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks, BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | Not reported | 55.62 ± 10.32% (best in column) | 100% (best in column) | Not reported | 30.0 ± 14.6% (best in column) | Not reported | 60.0 ± 20.3% (best in column) | Not reported | 1.60 ± 0.40 |
All 23 result rows with coverage, uncertainty and sources
Filter evaluations
Applied filters: All linked evaluations
| Tested configuration | Protocol and dataset | Finding | Evidence and details |
|---|---|---|---|
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench drug discovery domain: agent leaderboard percentile over 9 tasks Dataset subset: BioML-bench v0.1-alpha drug discovery tasks (9 tasks) | 30.6 ± 12.3% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 12.3 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench drug discovery (non-penalized) lit-agents-20261013-biomlbench-protocol-drug-discovery Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Drug Discovery', row 'Biomni', column 'Above Median (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench drug discovery domain: agent leaderboard percentile over 9 tasks Dataset subset: BioML-bench v0.1-alpha drug discovery tasks (9 tasks) | 13.9 ± 11.1% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 11.1 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench drug discovery (non-penalized) lit-agents-20261013-biomlbench-protocol-drug-discovery Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Drug Discovery', row 'Biomni', column 'Any Medal (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench drug discovery domain: agent leaderboard percentile over 9 tasks Dataset subset: BioML-bench v0.1-alpha drug discovery tasks (9 tasks) | 32.55 ± 8.61% Leaderboard percentile (mean over tasks in the domain; non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 8.61 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench drug discovery (non-penalized) lit-agents-20261013-biomlbench-protocol-drug-discovery Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Drug Discovery', row 'Biomni', column 'Leaderboard Percentile' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench drug discovery domain: agent leaderboard percentile over 9 tasks Dataset subset: BioML-bench v0.1-alpha drug discovery tasks (9 tasks) | 30.6 ± 12.3% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 12.3 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench drug discovery (penalized) lit-agents-20261013-biomlbench-protocol-drug-discovery Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Drug Discovery', row 'Biomni', column 'Above Median (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench drug discovery domain: agent leaderboard percentile over 9 tasks Dataset subset: BioML-bench v0.1-alpha drug discovery tasks (9 tasks) | 13.9 ± 11.1% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 11.1 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench drug discovery (penalized) lit-agents-20261013-biomlbench-protocol-drug-discovery Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Drug Discovery', row 'Biomni', column 'Any Medal (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench drug discovery domain: agent leaderboard percentile over 9 tasks Dataset subset: BioML-bench v0.1-alpha drug discovery tasks (9 tasks) | 100% Proportion (share of tasks that yield a valid, gradable submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: Not reported by the source Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench drug discovery (penalized) lit-agents-20261013-biomlbench-protocol-drug-discovery Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Drug Discovery', row 'Biomni', column 'Completion Rate (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench drug discovery domain: agent leaderboard percentile over 9 tasks Dataset subset: BioML-bench v0.1-alpha drug discovery tasks (9 tasks) | 32.55 ± 8.61% Leaderboard percentile (mean over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 8.61 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench drug discovery (penalized) lit-agents-20261013-biomlbench-protocol-drug-discovery Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Drug Discovery', row 'Biomni', column 'Leaderboard Percentile' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 25.0 ± 17.1% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 17.1 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench protein engineering (non-penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Protein Engineering', row 'Biomni', column 'Above Median (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 25.0 ± 17.1% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 17.1 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench protein engineering (non-penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Protein Engineering', row 'Biomni', column 'Any Medal (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 29.93 ± 11.04% Leaderboard percentile (mean over tasks in the domain; non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 11.04 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench protein engineering (non-penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Protein Engineering', row 'Biomni', column 'Leaderboard Percentile' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 25.0 ± 17.1% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 17.1 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench protein engineering (penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Protein Engineering', row 'Biomni', column 'Above Median (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 25.0 ± 17.1% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 17.1 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench protein engineering (penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Protein Engineering', row 'Biomni', column 'Any Medal (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 100% Proportion (share of tasks that yield a valid, gradable submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: Not reported by the source Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench protein engineering (penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Protein Engineering', row 'Biomni', column 'Completion Rate (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 29.93 ± 11.04% Leaderboard percentile (mean over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 11.04 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench protein engineering (penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Protein Engineering', row 'Biomni', column 'Leaderboard Percentile' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 60.0 ± 20.3% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 20.3 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Biomni', column 'Above Median (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 30.0 ± 14.6% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 14.6 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Biomni', column 'Any Medal (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 55.62 ± 10.32% Leaderboard percentile (mean over tasks in the domain; non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 10.32 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Biomni', column 'Leaderboard Percentile' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 2.20 ± 0.58 Rank (mean relative rank among the five Table 4 agents (Dummy, MLAgentBench, AIDE, STELLA, Biomni) by leaderboard percentile, over tasks in the domain; non-penalized: runs without a valid submission omitted) unitless · lower Uncertainty: ± 0.58 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Biomni', column 'Mean Rank' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 60.0 ± 20.3% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 20.3 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Single Cell Omics', row 'Biomni', column 'Above Median (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 30.0 ± 14.6% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 14.6 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Single Cell Omics', row 'Biomni', column 'Any Medal (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 100% Proportion (share of tasks that yield a valid, gradable submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: Not reported by the source Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Single Cell Omics', row 'Biomni', column 'Completion Rate (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 55.62 ± 10.32% Leaderboard percentile (mean over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 10.32 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Single Cell Omics', row 'Biomni', column 'Leaderboard Percentile' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 1.60 ± 0.40 Rank (mean relative rank among the five Table 4 agents (Dummy, MLAgentBench, AIDE, STELLA, Biomni) by leaderboard percentile, over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) unitless · lower Uncertainty: ± 0.40 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Single Cell Omics', row 'Biomni', column 'Mean Rank' |
Source checking is not independent reproduction. Release 2026-11-02-a7f03c9e55a5.
Use this configuration
How it works, versions and access
Additional methodological details, versions and access terms have not yet been extracted into a structured profile for this configuration. Consult the linked sources and evaluation records below.
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
2 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Parameters Biomni (specifically, v0.0.5) with Claude Sonnet 4 as LLM backend; agent class A1 in the repository wrapper (agents/biomni/main.py) Individual claims | science-machine/biomlbench repository at commit 673f3d8 Repository agents/biomni/config.yaml at commit 673f3d8, key biomni.kwargs.llm Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | conflicting claim ["source-hash-verification","independent-cell-check"] · 2026-10-13 Audit detailsIndependent review 2026-10-13: matched the cited repository file at commit 673f3d8. Conflicting source claim: Field: Claim: lit-agents-20261013-biomlbench-claim-biomni-repo-llm Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. Extraction artifact SHA-256: |
| Version v0.0.5 Individual claims | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Section 3.3 Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | conflicting claim ["source-hash-verification","pdf-text-parse","independent-cell-check"] · 2026-10-13 Audit detailsIndependent review 2026-10-13: wording found at the locator. Conflicting source claim: Field: Claim: lit-agents-20261013-biomlbench-claim-biomni-version Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. Extraction artifact SHA-256: |
Sources and history
Release 2026-11-02-a7f03c9e55a5 · Record review: source checked
2 source records and release history
- BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Original source · bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07)
- science-machine/biomlbench repository at commit 673f3d8 · Original source · Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)'
Technical metadata and extraction receipts
Stable ID: lit-agents-20261013-biomlbench-config-biomni-claude-sonnet-4
- areas
- molecular-interactions; protein-function; single-cell
- contexts
- research
- method types
- foundation_model
- reported name
- Biomni
- version
- v0.0.5
- parameters
- Biomni (specifically, v0.0.5) with Claude Sonnet 4 as LLM backend; agent class A1 in the repository wrapper (agents/biomni/main.py)
- source locator
- Section 3.3; repository agents/biomni/main.py line 11 ('from biomni.agent import A1')
- access reuse
- Open-source agent; LLM backend is a hosted API
- limitations
- Run by the BioML-bench authors in the configuration described in Section 3.3; four replicate runs per task.
- provenance note
- Agent configuration as run in BioML-bench (bioRxiv v3).
Related records
- configuration of: Biomni A1
- uses model: Claude Sonnet 4
- subject: config-biomni-claude-sonnet-4: attributes.parameters
- subject: config-biomni-claude-sonnet-4: attributes.version
- system: Biomni on BioML-bench drug discovery (non-penalized)
- system: Biomni on BioML-bench drug discovery (penalized)
- system: Biomni on BioML-bench protein engineering (non-penalized)
- system: Biomni on BioML-bench protein engineering (penalized)
- system: Biomni on BioML-bench single cell omics (non-penalized)
- system: Biomni on BioML-bench single cell omics (penalized)