The 5 single cell omics tasks of BioML-bench v0.1-alpha, as listed in Table 3: open-problems-modality, open-problems-perturbations, open-problems-label-projection, open-problems-spatial-genes, open-problems-cell-communication.
10 evaluations · 44 results. Different protocols are not a single leaderboard.
Applied filters: All linked evaluations
| Tested configuration | Protocol and dataset | Finding | Evidence and details |
|---|---|---|---|
| Configuration: AIDE v6.3.3, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 58.3 ± 16.7% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 16.7 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceAIDE on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'AIDE', column 'Above Median (%)' |
| Configuration: AIDE v6.3.3, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 33.3 ± 19.0% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 19.0 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceAIDE on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'AIDE', column 'Any Medal (%)' |
| Configuration: AIDE v6.3.3, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 60.67 ± 11.16% Leaderboard percentile (mean over tasks in the domain; non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 11.16 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceAIDE on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'AIDE', column 'Leaderboard Percentile' |
| Configuration: AIDE v6.3.3, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 2.30 ± 0.54 Rank (mean relative rank among the five Table 4 agents (Dummy, MLAgentBench, AIDE, STELLA, Biomni) by leaderboard percentile, over tasks in the domain; non-penalized: runs without a valid submission omitted) unitless · lower Uncertainty: ± 0.54 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceAIDE on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'AIDE', column 'Mean Rank' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 60.0 ± 20.3% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 20.3 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Biomni', column 'Above Median (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 30.0 ± 14.6% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 14.6 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Biomni', column 'Any Medal (%)' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 55.62 ± 10.32% Leaderboard percentile (mean over tasks in the domain; non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 10.32 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Biomni', column 'Leaderboard Percentile' |
| Configuration: Biomni v0.0.5, Claude Sonnet 4 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 2.20 ± 0.58 Rank (mean relative rank among the five Table 4 agents (Dummy, MLAgentBench, AIDE, STELLA, Biomni) by leaderboard percentile, over tasks in the domain; non-penalized: runs without a valid submission omitted) unitless · lower Uncertainty: ± 0.58 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceBiomni on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Biomni', column 'Mean Rank' |
| Configuration: Dummy agent (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 0.0 ± 0.0% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 0.0 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceDummy on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Dummy', column 'Above Median (%)' |
| Configuration: Dummy agent (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 0.0 ± 0.0% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 0.0 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceDummy on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Dummy', column 'Any Medal (%)' |
| Configuration: Dummy agent (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 7.34 ± 4.52% Leaderboard percentile (mean over tasks in the domain; non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 4.52 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceDummy on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Dummy', column 'Leaderboard Percentile' |
| Configuration: Dummy agent (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 4.80 ± 0.20 Rank (mean relative rank among the five Table 4 agents (Dummy, MLAgentBench, AIDE, STELLA, Biomni) by leaderboard percentile, over tasks in the domain; non-penalized: runs without a valid submission omitted) unitless · lower Uncertainty: ± 0.20 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceDummy on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'Dummy', column 'Mean Rank' |
| Configuration: MLAgentBench, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 37.5 ± 16.1% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 16.1 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceMLAgentBench on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'MLAgentBench', column 'Above Median (%)' |
| Configuration: MLAgentBench, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 31.2 ± 18.8% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 18.8 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceMLAgentBench on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'MLAgentBench', column 'Any Medal (%)' |
| Configuration: MLAgentBench, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 42.29 ± 13.78% Leaderboard percentile (mean over tasks in the domain; non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 13.78 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceMLAgentBench on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'MLAgentBench', column 'Leaderboard Percentile' |
| Configuration: MLAgentBench, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 3.50 ± 0.29 Rank (mean relative rank among the five Table 4 agents (Dummy, MLAgentBench, AIDE, STELLA, Biomni) by leaderboard percentile, over tasks in the domain; non-penalized: runs without a valid submission omitted) unitless · lower Uncertainty: ± 0.29 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceMLAgentBench on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'MLAgentBench', column 'Mean Rank' |
| Configuration: STELLA, OpenRouter default backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 68.3 ± 15.0% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 15.0 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceSTELLA on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'STELLA', column 'Above Median (%)' |
| Configuration: STELLA, OpenRouter default backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 46.7 ± 22.6% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 22.6 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceSTELLA on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'STELLA', column 'Any Medal (%)' |
| Configuration: STELLA, OpenRouter default backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 61.97 ± 13.91% Leaderboard percentile (mean over tasks in the domain; non-penalized: runs without a valid submission omitted) percent · higher Uncertainty: ± 13.91 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceSTELLA on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'STELLA', column 'Leaderboard Percentile' |
| Configuration: STELLA, OpenRouter default backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 1.90 ± 0.33 Rank (mean relative rank among the five Table 4 agents (Dummy, MLAgentBench, AIDE, STELLA, Biomni) by leaderboard percentile, over tasks in the domain; non-penalized: runs without a valid submission omitted) unitless · lower Uncertainty: ± 0.33 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceSTELLA on BioML-bench single cell omics (non-penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Non-penalized', domain 'Single Cell Omics', row 'STELLA', column 'Mean Rank' |
| Configuration: AIDE v6.3.3, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 55.0 ± 16.6% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 16.6 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceAIDE on BioML-bench single cell omics (penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Single Cell Omics', row 'AIDE', column 'Above Median (%)' |
| Configuration: AIDE v6.3.3, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 35.0 ± 18.7% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 18.7 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceAIDE on BioML-bench single cell omics (penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Single Cell Omics', row 'AIDE', column 'Any Medal (%)' |
| Configuration: AIDE v6.3.3, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 85% Proportion (share of tasks that yield a valid, gradable submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: Not reported by the source Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceAIDE on BioML-bench single cell omics (penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Single Cell Omics', row 'AIDE', column 'Completion Rate (%)' |
| Configuration: AIDE v6.3.3, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 53.17 ± 14.86% Leaderboard percentile (mean over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 14.86 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceAIDE on BioML-bench single cell omics (penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Single Cell Omics', row 'AIDE', column 'Leaderboard Percentile' |
| Configuration: AIDE v6.3.3, GPT-4.1 backend (BioML-bench) | Protocol: BioML-bench single cell omics domain: agent leaderboard percentile over 5 tasks Dataset subset: BioML-bench v0.1-alpha single cell omics tasks (5 tasks) | 2.70 ± 0.44 Rank (mean relative rank among the five Table 4 agents (Dummy, MLAgentBench, AIDE, STELLA, Biomni) by leaderboard percentile, over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) unitless · lower Uncertainty: ± 0.44 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceAIDE on BioML-bench single cell omics (penalized) lit-agents-20261013-biomlbench-protocol-single-cell-omics Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Single Cell Omics', row 'AIDE', column 'Mean Rank' |
Source checking is not independent reproduction. Release 2026-11-02-a7f03c9e55a5.
0 of 4 readiness checks met. These checks assess whether the evidence supports a reproducible investigation; a source-checked score alone does not meet them.
Release 2026-11-02-a7f03c9e55a5 · Evidence verified: Not verified
Exact outcomes, predictions, identifiers and evaluator are connected.
Missing or unresolved evidence
Verified: Not verified
Replay evidence includes annotations and an assessment of dependence. Unknown independence permits descriptive analysis only.
Missing or unresolved evidence
Verified: Not verified
A pinned recipe describes the inputs, environment and resource requirements.
Missing or unresolved evidence
Verified: Not verified
Separate data and exposure records support an independent test.
Missing or unresolved evidence
Verified: Not verified
Readiness describes the evidence in this release. Availability on your computer is checked separately when an investigation runs. Existing data exposure can prevent independent validation even when files are available.
No verified artifact manifest is connected to this record yet. The gaps above identify what is needed before analysis can begin.
This record describes a particular subset or cohort used in an evaluation. Its results do not describe the full dataset.
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
12 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Source location Table 3; repository experiments/biomlbench_v0.1a.txt Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 3; repository experiments/biomlbench_v0.1a.txt Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Source location Table 3; repository experiments/biomlbench_v0.1a.txt Context-only references | science-machine/biomlbench repository at commit 673f3d8 Table 3; repository experiments/biomlbench_v0.1a.txt Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. |
| Split Task-specific splits defined by BioML-bench's own preparation scripts Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 3; repository experiments/biomlbench_v0.1a.txt Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Split Task-specific splits defined by BioML-bench's own preparation scripts Context-only references | science-machine/biomlbench repository at commit 673f3d8 Table 3; repository experiments/biomlbench_v0.1a.txt Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. |
| Subset scope 5 tasks listed under 'Single-cell omics' in Table 3. Open Problems (paper Table 1). BioML-bench prepares its own data files and splits; they are not the Open Problems benchmark splits. Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 3; repository experiments/biomlbench_v0.1a.txt Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Subset scope 5 tasks listed under 'Single-cell omics' in Table 3. Open Problems (paper Table 1). BioML-bench prepares its own data files and splits; they are not the Open Problems benchmark splits. Context-only references | science-machine/biomlbench repository at commit 673f3d8 Table 3; repository experiments/biomlbench_v0.1a.txt Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. |
| Version BioML-bench v0.1-alpha task set (experiments/biomlbench_v0.1a.txt at commit 673f3d8) Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 3; repository experiments/biomlbench_v0.1a.txt Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Version BioML-bench v0.1-alpha task set (experiments/biomlbench_v0.1a.txt at commit 673f3d8) Context-only references | science-machine/biomlbench repository at commit 673f3d8 Table 3; repository experiments/biomlbench_v0.1a.txt Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. |
| Description The 5 single cell omics tasks of BioML-bench v0.1-alpha, as listed in Table 3: open-problems-modality, open-problems-perturbations, open-problems-label-projection, open-problems-spatial-genes, open-problems-cell-communication. Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 3; repository experiments/biomlbench_v0.1a.txt Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Description The 5 single cell omics tasks of BioML-bench v0.1-alpha, as listed in Table 3: open-problems-modality, open-problems-perturbations, open-problems-label-projection, open-problems-spatial-genes, open-problems-cell-communication. Context-only references | science-machine/biomlbench repository at commit 673f3d8 Table 3; repository experiments/biomlbench_v0.1a.txt Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 673f3d8d3462fc6cdcbbd9695fba612b9332ca65 (2025-09-14, head of main at retrieval); README title 'BioML-bench (v0.1-alpha)' | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the codeload tar.gz archive for the commit; two downloads gave identical bytes. Per-file hashes of the files cited are in sources.md. |
Release 2026-11-02-a7f03c9e55a5 · Record review: source checked
Stable ID: lit-agents-20261013-biomlbench-data-single-cell-omics-tasks