Evaluation

Training-mean control on mRNABench Sample designed MRL

Predict target_mrl_designed from sequence on the complete 15,003-record canonical test split. Train-only RidgeCV differs from upstream default validation evaluation; this is not an aggregate mRNABench score.

Research readiness

These checks assess whether the evidence supports a reproducible investigation. A source-checked score alone does not meet these requirements.

Release 2026-09-29-06401fd5b220 · Evidence verified: 2026-09-23

Evidence complete

Replay metrics

Exact outcomes, predictions, identifiers and evaluator are connected.

Supporting evidence

mrnabench-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
mrnabench-discrepancies: join_integrity: 15003 unique test IDs; scores and outcomes reconcile.
mrnabench-discrepancies: score_semantics: Higher scores predict higher mean ribosome load; metrics retain their own direction.
mrnabench-discrepancies: metric_replay: Saved predictions reproduce recorded metrics within 1e-9 absolute tolerance.

Verified: 2026-09-23

Evidence complete

Investigate discrepancies

Replay evidence includes annotations and an assessment of dependence. Unknown independence permits descriptive analysis only.

Supporting evidence

mrnabench-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
mrnabench-discrepancies: join_integrity: 15003 unique test IDs; scores and outcomes reconcile.
mrnabench-discrepancies: score_semantics: Higher scores predict higher mean ribosome load; metrics retain their own direction.
mrnabench-discrepancies: metric_replay: Saved predictions reproduce recorded metrics within 1e-9 absolute tolerance.
mrnabench-discrepancies: annotations: Registered subgroup fields derive from the preserved source snapshot.
mrnabench-discrepancies: dependence: Independent sampling unit unknown; descriptive analysis only.

Verified: 2026-09-23

Evidence complete

Run locally

A pinned recipe describes the inputs, environment and resource requirements.

Supporting evidence

mrnabench-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
mrnabench-discrepancies: join_integrity: 15003 unique test IDs; scores and outcomes reconcile.
mrnabench-discrepancies: score_semantics: Higher scores predict higher mean ribosome load; metrics retain their own direction.
mrnabench-discrepancies: recipe_pinned: Registered SDK recipe; implementation artifact and environment hashes retained.
mrnabench-discrepancies: resource_estimate: Local CPU controls previously executed; all jobs remain subject to the campaign watchdog.

Verified: 2026-09-23

Evidence incomplete

Validate independently

Separate data and exposure records support an independent test.

Missing or unresolved evidence

independent validation: verification is missing
overlap checked: verification is missing
Data has already been exposed during hypothesis selection; it is not untouched validation.

Supporting evidence

mrnabench-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
mrnabench-discrepancies: join_integrity: 15003 unique test IDs; scores and outcomes reconcile.
mrnabench-discrepancies: score_semantics: Higher scores predict higher mean ribosome load; metrics retain their own direction.

Verified: 2026-09-23

Readiness describes the evidence in this release. Availability on your computer is checked separately when an investigation runs. Existing data exposure can prevent independent validation even when files are available.

Limitations

Existing test outcomes are exposed; no independent validation is claimed.
Independent experimental grouping is unavailable; subgroup summaries are descriptive.
Prepared opaque IDs must not be joined to a newly prepared snapshot by ID.
Random split does not establish homology separation.
Source data reuse licence is unreported; no raw data redistribution.
The sample independence unit is unresolved; descriptive analysis does not establish confirmatory inference.
Evidence is exploratory because this data has already been examined.

Artifacts and reproduction

One or more artifacts need a local resolver. Their checksums are recorded, but this release does not provide every download.

mRNABench designed: composition effects

Where does the composition predictor improve over the training-mean control?

Manifest: mrnabench-discrepancies
Prepared outcome: mean ribosome load
Outcome type: continuous
Units: MRL
Score direction: higher
Join identifier: id
Unit of independence: Not reported
Split: test
Prior data exposure: Already used for exploration; not untouched validation

Evaluation protocol and run instructions

Resolve each required artifact from its recorded source, verify its SHA-256, and retain the prepared snapshot and identifiers. A missing public URL means this release does not redistribute that file.

mrnabench-discrepancies-prepared

prepared · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: e5d6fb9e5fadec1f9e6235ca3ed2054ac05f476ccc8f5dc67ec500febeae8237

Semantic SHA-256: f4f9777b5119b739437c6a9d85e4e1b937268dafd1307efca5f072d38957738b (separate from the file checksum)

mrnabench-discrepancies-mrnabench-composition-predictions

mrnabench-composition-predictions · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: a8f80cfea7ddd197ef9446e161f5a761741b3f778dfc1459275c3355432e15c5

mrnabench-discrepancies-mrnabench-composition-report

mrnabench-composition-report · json

Artifact source

File SHA-256: 8b68dfe5725bf5d8c108265039e3787f4b3b55986eab8a430082c4d446a4f960

mrnabench-discrepancies-mrnabench-composition-source

mrnabench-composition-source · json

Artifact source

File SHA-256: ef486f7be79a6f5589afd7494b7acfa13ca698da50f0ffd0e50fd6eba1eb9934

mrnabench-discrepancies-mrnabench-train-mean-predictions

mrnabench-train-mean-predictions · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: fb435ffda2e56792a9c781fba8dfda535b3d6508cf28f484162c5fd9065a87d8

mrnabench-discrepancies-mrnabench-train-mean-report

mrnabench-train-mean-report · json

Artifact source

File SHA-256: c036d9fa75b6ba4d5c65d31cd575ee2bfc357beec3ca7e6c80f07e3128a04e71

mrnabench-discrepancies-mrnabench-train-mean-source

mrnabench-train-mean-source · json

Artifact source

File SHA-256: b8bbc4f3f2ef54d92721015205bdd73a81e4cfcbc596985e8724be3416899d54

mrnabench-discrepancies-recipe_code

recipe code · py

Artifact source

File SHA-256: b3407940ca4c8a1cdcf3a3bb3849c3e26d7f85272b2ee63087307378ffaa79dc

mrnabench-discrepancies-table

table · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: d49ff1b4dacdfe0663c35bdad6a9ad91e79f5862fe96f36a627acb8407610bf3

mrnabench-discrepancies-receipt

receipt · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 5edeb10e42be225f6b85481c95eb8985ab1724d3af353737fdafd29ccdfc4022

mrnabench-discrepancies-preparation_code

preparation code · py

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 8a1d691de8fdd9fcf8242fe6fa2cfe2703a1c3281b880fe2a226cb8bd946fd1b

mrnabench-discrepancies-environment

environment · lock

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 748257abb3664595db1ebf892799e91c6f1a4ca79b0e65eccb3958cf5644a671

Registered recipes and verification receipt

Local recipes: sdk:train-mean-v1; sdk:sequence-composition-v1
Verification: verified at: 2026-09-23T23:00:14.349323+00:00; checks: check: artifact_hashes; status: passed; detail: Exact local bytes recorded; source pins checked where available.; check: join_integrity; status: passed; detail: 15003 unique test IDs; scores and outcomes reconcile.; check: score_semantics; status: passed; detail: Higher scores predict higher mean ribosome load; metrics retain their own direction.; check: metric_replay; status: passed; detail: Saved predictions reproduce recorded metrics within 1e-9 absolute tolerance.; check: annotations; status: passed; detail: Registered subgroup fields derive from the preserved source snapshot.; check: dependence; status: passed; detail: Independent sampling unit unknown; descriptive analysis only.; check: recipe_pinned; status: passed; detail: Registered SDK recipe; implementation artifact and environment hashes retained.; check: resource_estimate; status: passed; detail: Local CPU controls previously executed; all jobs remain subject to the campaign watchdog.; limitations: Existing test outcomes are exposed; no independent validation is claimed.; Independent experimental grouping is unavailable; subgroup summaries are descriptive.; Prepared opaque IDs must not be joined to a newly prepared snapshot by ID.; Random split does not establish homology separation.; Source data reuse licence is unreported; no raw data redistribution.

Read reviewed discrepancy investigations

Evaluation results

1 evaluation · 3 results. Different protocols are not a single leaderboard.

Filter evaluations

Tested model or methodProtocolDatasetEvaluation setupMetricEvidence origin

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configuration	Protocol and dataset	Finding	Evidence and details
Configuration: Training-mean control (mRNABench Sample designed MRL)	Protocol: mRNABench Sample designed MRL Dataset subset: mRNABench Sample designed MRL test subset	2.37 mse squared mean ribosome load · lower Uncertainty: Not reported Coverage: 15003/15003	Rewire evaluation · Source checked Methods, coverage and source Training-mean control on mRNABench Sample designed MRL mRNABench Sample designed MRL Aggregation: Not reported Training-mean control: local execution report (20 September 2026); Training-mean control: automated execution audit · metrics.mse
Configuration: Training-mean control (mRNABench Sample designed MRL)	Protocol: mRNABench Sample designed MRL Dataset subset: mRNABench Sample designed MRL test subset	undefined pearson dimensionless · higher Uncertainty: Not reported Coverage: 15003/15003	Rewire evaluation · Source checked Methods, coverage and source Training-mean control on mRNABench Sample designed MRL mRNABench Sample designed MRL Aggregation: Not reported Training-mean control: local execution report (20 September 2026); Training-mean control: automated execution audit · metrics.pearson
Configuration: Training-mean control (mRNABench Sample designed MRL)	Protocol: mRNABench Sample designed MRL Dataset subset: mRNABench Sample designed MRL test subset	undefined spearman dimensionless · higher Uncertainty: Not reported Coverage: 15003/15003	Rewire evaluation · Source checked Methods, coverage and source Training-mean control on mRNABench Sample designed MRL mRNABench Sample designed MRL Aggregation: Not reported Training-mean control: local execution report (20 September 2026); Training-mean control: automated execution audit · metrics.spearman

Source checking is not independent reproduction. Release 2026-09-29-06401fd5b220.

Evaluation procedure

mRNABench Sample designed MRL

Configuration: Training-mean control (mRNABench Sample designed MRL)
Protocol: mRNABench Sample designed MRL
Dataset subset: mRNABench Sample designed MRL test subset

origin: Rewire evaluation
configuration: 1
protocol id: rewire-protocol-mrnabench-designed-mrl-v1
dataset version: rewire-dataset-mrnabench-designed-mrl-v1
split: canonical test
subset: mrnabench-sample-mrl-designed-target_mrl_designed
population: 15003/15003
inputs: Complete source sequence only; no assay labels during prediction
adaptation: train-only fitting
metric implementation: fcc5ab65e6753cf29aabbdae231cbb5cdf72a3dd40ba8f109802c33223b33536
aggregation: all held-out test rows of the selected target
budget: one local CPU evaluation; no hyperparameter search outside training

Metadata review: source checked. Unreported conditions prevent automatic comparisons.

Reproduction

Split: canonical test
Adaptation: train-only fitting
Scoring implementation: fcc5ab65e6753cf29aabbdae231cbb5cdf72a3dd40ba8f109802c33223b33536

Generate and evaluate predictions

This committed script regenerates the selected local evaluation with the recorded inputs and configuration. Sequence-control instructions execute all four controls; select this evaluation from their outputs. Raw prior predictions are private, so public evidence alone cannot rescore them.

mRNABench Sample designed MRL: reproduction instructions · Reproduce section

Access requirements and official instructions: mRNABench Sample designed MRL

Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

132 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-29-06401fd5b220
Property and statement	Original source and location	Review and provenance
attributes.comparison.adaptation train-only fitting Context-only references	mRNABench Sample designed MRL: reproduction instructions Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.adaptation` Source artifact SHA-256: `75f50703210dc556d77fdfe8f1d28c3200121b5b78c5c6592a096d928b4ff7cc` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.adaptation train-only fitting Context-only references	Training-mean control: local execution report (20 September 2026) Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.adaptation` Source artifact SHA-256: `c036d9fa75b6ba4d5c65d31cd575ee2bfc357beec3ca7e6c80f07e3128a04e71` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.aggregation all held-out test rows of the selected target Context-only references	mRNABench Sample designed MRL: reproduction instructions Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.aggregation` Source artifact SHA-256: `75f50703210dc556d77fdfe8f1d28c3200121b5b78c5c6592a096d928b4ff7cc` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.aggregation all held-out test rows of the selected target Context-only references	Training-mean control: local execution report (20 September 2026) Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.aggregation` Source artifact SHA-256: `c036d9fa75b6ba4d5c65d31cd575ee2bfc357beec3ca7e6c80f07e3128a04e71` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.budget one local CPU evaluation; no hyperparameter search outside training Context-only references	mRNABench Sample designed MRL: reproduction instructions Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.budget` Source artifact SHA-256: `75f50703210dc556d77fdfe8f1d28c3200121b5b78c5c6592a096d928b4ff7cc` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.budget one local CPU evaluation; no hyperparameter search outside training Context-only references	Training-mean control: local execution report (20 September 2026) Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.budget` Source artifact SHA-256: `c036d9fa75b6ba4d5c65d31cd575ee2bfc357beec3ca7e6c80f07e3128a04e71` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.dataset_version rewire-dataset-mrnabench-designed-mrl-v1 Context-only references	mRNABench Sample designed MRL: reproduction instructions Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.dataset_version` Source artifact SHA-256: `75f50703210dc556d77fdfe8f1d28c3200121b5b78c5c6592a096d928b4ff7cc` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.dataset_version rewire-dataset-mrnabench-designed-mrl-v1 Context-only references	Training-mean control: local execution report (20 September 2026) Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.dataset_version` Source artifact SHA-256: `c036d9fa75b6ba4d5c65d31cd575ee2bfc357beec3ca7e6c80f07e3128a04e71` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.inputs Complete source sequence only; no assay labels during prediction Context-only references	mRNABench Sample designed MRL: reproduction instructions Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.inputs` Source artifact SHA-256: `75f50703210dc556d77fdfe8f1d28c3200121b5b78c5c6592a096d928b4ff7cc` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.inputs Complete source sequence only; no assay labels during prediction Context-only references	Training-mean control: local execution report (20 September 2026) Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.inputs` Source artifact SHA-256: `c036d9fa75b6ba4d5c65d31cd575ee2bfc357beec3ca7e6c80f07e3128a04e71` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

View linked audit checks and correction history

Release 2026-09-29-06401fd5b220 · Record review: source checked

2 source records and release history

Training-mean control: local execution report (20 September 2026) · Original source · ca73fa47136d182f2d4ddb083d084712198fc0e2
mRNABench Sample designed MRL: reproduction instructions · Original source · ca73fa47136d182f2d4ddb083d084712198fc0e2

Download this release

Technical metadata and extraction receipts

Stable ID: rewire-local-20260920-evaluation-mrnabench-train-mean

areas: rna-transcriptomics
origin: rewire_run
execution scope: complete_selected_evaluation
published score reproduction: false
protocol: mRNABench Sample designed MRL
version: 1
source locator: metrics; coverage; protocol_results
original sdk scope: full
original sdk completion: complete
eligible count: 15003
scored count: 15003
missing count: 0
coverage: 15003/15003
suite complete: false
comparison: protocol id: rewire-protocol-mrnabench-designed-mrl-v1; dataset version: rewire-dataset-mrnabench-designed-mrl-v1; split: canonical test; subset: mrnabench-sample-mrl-designed-target_mrl_designed; population: 15003/15003; inputs: Complete source sequence only; no assay labels during prediction; adaptation: train-only fitting; metric implementation: fcc5ab65e6753cf29aabbdae231cbb5cdf72a3dd40ba8f109802c33223b33536; aggregation: all held-out test rows of the selected target; budget: one local CPU evaluation; no hyperparameter search outside training
input information: Complete source sequence only; no assay labels during prediction
model configuration: constant: arithmetic mean of training targets; fit split: train; validation used: false
provenance: data verification: pinned_source_bytes; dataset revision: ef67f7cf8a999bb1c412ad6551aa7d9f901cbb95; source license: unreported; source sha256: 8b8c57581d472e89a21f8b468b3b0599f6c235c01423264561e6c302d78927d4; source url: https://huggingface.co/datasets/morrislab/mrl-sample/resolve/ef67f7cf8a999bb1c412ad6551aa7d9f901cbb95/mrl-sample-designed.parquet; split sha256: 00e287b18ba4e90cb682af6e3fcdfc97e07f2b4b25d13265cd0eca37996536a2; upstream code license: AGPL-3.0; upstream revision: 74f96b8e6ae9f41cc3cccff089d826a62d5604b8
predictions sha256: fb435ffda2e56792a9c781fba8dfda535b3d6508cf28f484162c5fd9065a87d8
prepared sha256: f4f9777b5119b739437c6a9d85e4e1b937268dafd1307efca5f072d38957738b
execution: adapter: TrainMean; batch size: 32; fitting: train_only; inference and fit seconds: 0.06764833299530437; mode: local_adapter; prediction type: scalar
timing seconds: evaluation: 1.1848423330011428
environment: architecture: arm64; container digest: unreported; container runtime: unreported; platform: macOS-26.6.2-arm64-arm-64bit; python: 3.11.13; rewirebench: 0.4.0; sdk code sha256: fcc5ab65e6753cf29aabbdae231cbb5cdf72a3dd40ba8f109802c33223b33536; sif sha256: unreported
run url: https://github.com/rewire-bio/rewire-benchmarks/blob/ca73fa47136d182f2d4ddb083d084712198fc0e2/research/local-runs-2026-09-20/mrnabench-train-mean/report.json
reproduction url: https://github.com/rewire-bio/rewire-benchmarks/blob/ca73fa47136d182f2d4ddb083d084712198fc0e2/research/local-runs-2026-09-20/README.md
limitations: One selected split or assay; no whole-suite score.; No published model score reproduction claim.; Raw predictions retained locally; hashes and inspected contribution bundle are public.; Single execution; no seed variability or uncertainty interval estimated.
reproduction: recipe owner id: rewire-protocol-mrnabench-designed-mrl-v1; recipe id: local-20260920-mrnabench-repeat; applicability: generate_and_evaluate; explanation: This committed script regenerates the selected local evaluation with the recorded inputs and configuration. Sequence-control instructions execute all four controls; select this evaluation from their outputs. Raw prior predictions are private, so public evidence alone cannot rescore them.; source ids: rewire-local-20260920-instructions-mrnabench; source locator: Reproduce section

Related records

configuration: Training-mean control (mRNABench Sample designed MRL)
protocol: mRNABench Sample designed MRL
dataset subset: mRNABench Sample designed MRL test subset
evaluation: Training-mean control: mse
evaluation: Training-mean control: pearson
evaluation: Training-mean control: spearman

Suggest a correction