Published comparison transcribed from the pinned source; not reproduced.
1 evaluation · 4 results. Different protocols are not a single leaderboard.
Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks · Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks)
Sorted by Leaderboard percentile (mean over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) (higher is better). The best value in each column is highlighted. Decimals are rounded for display; each value links to the printed value and its source.
| Tested configuration | Leaderboard percentile (mean over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) | Proportion (share of tasks that yield a valid, gradable submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) | Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); penalized: runs without a valid submission scored as percentile 0) | Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) |
|---|---|---|---|---|
| STELLA, OpenRouter default backend (BioML-bench)Evaluation | 34.98 ± 12.51% | 100% | 16.7 ± 12.4% | 45.8 ± 18.7% |
Applied filters: All linked evaluations
| Tested configuration | Protocol and dataset | Finding | Evidence and details |
|---|---|---|---|
| Configuration: STELLA, OpenRouter default backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 45.8 ± 18.7% Proportion (share of tasks where the agent's mean score exceeds the median human leaderboard submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 18.7 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceSTELLA on BioML-bench protein engineering (penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Protein Engineering', row 'STELLA', column 'Above Median (%)' |
| Configuration: STELLA, OpenRouter default backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 16.7 ± 12.4% Proportion (share of tasks where the agent reaches a public leaderboard medal threshold, MLE-bench definition (Section 3.4); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 12.4 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceSTELLA on BioML-bench protein engineering (penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Protein Engineering', row 'STELLA', column 'Any Medal (%)' |
| Configuration: STELLA, OpenRouter default backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 100% Proportion (share of tasks that yield a valid, gradable submission (Section 3.4 definition); penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: Not reported by the source Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceSTELLA on BioML-bench protein engineering (penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Protein Engineering', row 'STELLA', column 'Completion Rate (%)' |
| Configuration: STELLA, OpenRouter default backend (BioML-bench) | Protocol: BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks Dataset subset: BioML-bench v0.1-alpha protein engineering tasks (6 tasks) | 34.98 ± 12.51% Leaderboard percentile (mean over tasks in the domain; penalized: runs without a valid submission scored as percentile 0) percent · higher Uncertainty: ± 12.51 (type not stated). Table 4 caption does not name the spread. Figures 2 and 4 show per-domain leaderboard percentiles as mean ± SEM across tasks; the Table 2 caption refers to the reported SEM. Coverage: Not reported scored / Not reported eligible | Independent external evaluation · Source checkedMethods, coverage and sourceSTELLA on BioML-bench protein engineering (penalized) lit-agents-20261013-biomlbench-protocol-protein-engineering Aggregation: Not reported BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML · Table 4 (page 12), block 'Penalized', domain 'Protein Engineering', row 'STELLA', column 'Leaderboard Percentile' |
Source checking is not independent reproduction. Release 2026-11-02-a7f03c9e55a5.
0 of 4 readiness checks met. These checks assess whether the evidence supports a reproducible investigation; a source-checked score alone does not meet them.
Release 2026-11-02-a7f03c9e55a5 · Evidence verified: Not verified
Exact outcomes, predictions, identifiers and evaluator are connected.
Missing or unresolved evidence
Verified: Not verified
Replay evidence includes annotations and an assessment of dependence. Unknown independence permits descriptive analysis only.
Missing or unresolved evidence
Verified: Not verified
A pinned recipe describes the inputs, environment and resource requirements.
Missing or unresolved evidence
Verified: Not verified
Separate data and exposure records support an independent test.
Missing or unresolved evidence
Verified: Not verified
Readiness describes the evidence in this release. Availability on your computer is checked separately when an investigation runs. Existing data exposure can prevent independent validation even when files are available.
No verified artifact manifest is connected to this record yet. The gaps above identify what is needed before analysis can begin.
BioML-bench protein engineering domain: agent leaderboard percentile over 6 tasks
Metadata review: source checked. Unreported conditions prevent automatic comparisons.
No execution recipe has been verified for this exact configuration and evaluation. A benchmark's general instructions may use different inputs, splits or model settings.
Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
20 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Comparison: adaptation Autonomous agent builds and trains its own ML pipeline; LLM backend: OpenRouter default (model not named) Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 4, block 'Penalized', domain 'Protein Engineering', row 'STELLA' Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded independent paper Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Comparison: aggregation Mean over tasks in the domain; penalized: runs without a valid submission scored as percentile 0 Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 4, block 'Penalized', domain 'Protein Engineering', row 'STELLA' Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded independent paper Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Comparison: budget CPU-only, 16 vCPUs, 64 GB RAM, 8-hour wall-clock limit per run Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 4, block 'Penalized', domain 'Protein Engineering', row 'STELLA' Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded independent paper Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Comparison: dataset version BioML-bench v0.1-alpha task set (Table 3) Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 4, block 'Penalized', domain 'Protein Engineering', row 'STELLA' Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded independent paper Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Comparison: inputs Task description, training data, test features and sample submission Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 4, block 'Penalized', domain 'Protein Engineering', row 'STELLA' Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded independent paper Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Comparison: metric implementation Task's original metric mapped to the task's public leaderboard percentile Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 4, block 'Penalized', domain 'Protein Engineering', row 'STELLA' Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded independent paper Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Comparison: population 6 protein engineering tasks, four replicate runs per task Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 4, block 'Penalized', domain 'Protein Engineering', row 'STELLA' Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded independent paper Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Comparison: protocol id lit-agents-20261013-biomlbench-protocol-protein-engineering Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 4, block 'Penalized', domain 'Protein Engineering', row 'STELLA' Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded independent paper Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
| Comparison: split ProteinGym supervised cross-validation folds (fold_random_5, fold_modulo_5, fold_contiguous_5) per DMS assay Context-only references | BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 4, block 'Penalized', domain 'Protein Engineering', row 'STELLA' Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded independent paper Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
Limitations1 values
| BioML-bench: Evaluation of AI Agents for End-to-End Biomedical ML Table 4, block 'Penalized', domain 'Protein Engineering', row 'STELLA' Version: bioRxiv 2025.09.01.673319 version 3, posted 2025-09-28 (versions 1 and 2 posted 2025-09-04 and 2025-09-07) | not individually reviewed No individual claim review recorded independent paper Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes as downloaded 2026-10-13T12:38:42Z. bioRxiv rewrites the XMP dates and document IDs, the Info ModDate and the trailer /ID on download, so a re-download does not reproduce the hash; the review found a fresh copy differing only in those 43 bytes, with an identical text layer. |
Release 2026-11-02-a7f03c9e55a5 · Record review: source checked
Stable ID: lit-agents-20261013-biomlbench-eval-protein-engineering-penalized-stella