Evaluation

Training-mean control on FLIP2 Rhomax by_wild_type

Predict measured Rhomax wavelengths (nm) from amino-acid sequence on all 184 held-out archived test records. This selected split is not a complete FLIP2 score.

Research readiness

These checks assess whether the evidence supports a reproducible investigation. A source-checked score alone does not meet these requirements.

Release 2026-09-29-06401fd5b220 · Evidence verified: 2026-09-23

Evidence complete

Replay metrics

Exact outcomes, predictions, identifiers and evaluator are connected.

Supporting evidence

flip2-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
flip2-discrepancies: join_integrity: 184 unique test IDs; scores and outcomes reconcile.
flip2-discrepancies: score_semantics: Higher scores predict higher spectral wavelength; metrics retain their own direction.
flip2-discrepancies: metric_replay: Saved predictions reproduce recorded metrics within 1e-9 absolute tolerance.

Verified: 2026-09-23

Evidence complete

Investigate discrepancies

Replay evidence includes annotations and an assessment of dependence. Unknown independence permits descriptive analysis only.

Supporting evidence

flip2-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
flip2-discrepancies: join_integrity: 184 unique test IDs; scores and outcomes reconcile.
flip2-discrepancies: score_semantics: Higher scores predict higher spectral wavelength; metrics retain their own direction.
flip2-discrepancies: metric_replay: Saved predictions reproduce recorded metrics within 1e-9 absolute tolerance.
flip2-discrepancies: annotations: Registered subgroup fields derive from the preserved source snapshot.
flip2-discrepancies: dependence: Independent sampling unit unknown; descriptive analysis only.

Verified: 2026-09-23

Evidence complete

Run locally

A pinned recipe describes the inputs, environment and resource requirements.

Supporting evidence

flip2-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
flip2-discrepancies: join_integrity: 184 unique test IDs; scores and outcomes reconcile.
flip2-discrepancies: score_semantics: Higher scores predict higher spectral wavelength; metrics retain their own direction.
flip2-discrepancies: recipe_pinned: Registered SDK recipe; implementation artifact and environment hashes retained.
flip2-discrepancies: resource_estimate: Local CPU controls previously executed; all jobs remain subject to the campaign watchdog.

Verified: 2026-09-23

Evidence incomplete

Validate independently

Separate data and exposure records support an independent test.

Missing or unresolved evidence

independent validation: verification is missing
overlap checked: verification is missing
Data has already been exposed during hypothesis selection; it is not untouched validation.

Supporting evidence

flip2-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
flip2-discrepancies: join_integrity: 184 unique test IDs; scores and outcomes reconcile.
flip2-discrepancies: score_semantics: Higher scores predict higher spectral wavelength; metrics retain their own direction.

Verified: 2026-09-23

Readiness describes the evidence in this release. Availability on your computer is checked separately when an investigation runs. Existing data exposure can prevent independent validation even when files are available.

Limitations

Existing test outcomes are exposed; no independent validation is claimed.
Independent experimental grouping is unavailable; subgroup summaries are descriptive.
Prepared opaque IDs must not be joined to a newly prepared snapshot by ID.
High wavelength is not universally biologically preferable; NDCG is a numerical ranking metric.
Known methodological control, not a novel biological finding.
The sample independence unit is unresolved; descriptive analysis does not establish confirmatory inference.
Evidence is exploratory because this data has already been examined.

Artifacts and reproduction

One or more artifacts need a local resolver. Their checksums are recorded, but this release does not provide every download.

FLIP2 Rhomax: metric interpretation control

Why can a constant predictor have high NDCG and undefined Spearman correlation?

Manifest: flip2-discrepancies
Prepared outcome: spectral wavelength
Outcome type: continuous
Units: nm
Score direction: higher
Join identifier: id
Unit of independence: Not reported
Split: test
Prior data exposure: Already used for exploration; not untouched validation

Evaluation protocol and run instructions

Resolve each required artifact from its recorded source, verify its SHA-256, and retain the prepared snapshot and identifiers. A missing public URL means this release does not redistribute that file.

flip2-discrepancies-prepared

prepared · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: df996aee84bd5ed3e6381f9f6db864a377ab0b4809a9a48ff146d06a3dc069df

Semantic SHA-256: f417d43497e0c8c453bd7c66cfea05959b4d6d47462283096e42a2dae8b72419 (separate from the file checksum)

flip2-discrepancies-flip2-composition-predictions

flip2-composition-predictions · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 72ea3c506b8645ae6020207f904fbd5c0b09549cd125fcdcabfb487e68681b9c

flip2-discrepancies-flip2-composition-report

flip2-composition-report · json

Artifact source

File SHA-256: abd08d3f493428d06f43740c94e268575d755560e72120c73b294127acb395e6

flip2-discrepancies-flip2-composition-source

flip2-composition-source · json

Artifact source

File SHA-256: 7157313eab75ff8f93b934ac186852c8c82f8bb885617cab70c89427c631c1e4

flip2-discrepancies-flip2-train-mean-predictions

flip2-train-mean-predictions · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 599dca4e6fa643a7cdfc96ac339033f61e5846ce225af17ac2ac3917ecc14fac

flip2-discrepancies-flip2-train-mean-report

flip2-train-mean-report · json

Artifact source

File SHA-256: eb89260f6e138bb9ae2fdf3b882a87fdf8aeabd54a0ac6f17a73f5671c96b4d8

flip2-discrepancies-flip2-train-mean-source

flip2-train-mean-source · json

Artifact source

File SHA-256: 8bd712289f136edbfde3fa885166ef85e4f6fe3c4c437810c24dadb87107cc3f

flip2-discrepancies-recipe_code

recipe code · py

Artifact source

File SHA-256: b3407940ca4c8a1cdcf3a3bb3849c3e26d7f85272b2ee63087307378ffaa79dc

flip2-discrepancies-table

table · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 8562d4891270a73026fc7852c05f41206ad52e8fcae1d706d83ec57572c7c949

flip2-discrepancies-receipt

receipt · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 68631e653cd5aa9047b3f4e88cbb3fec94630847c1b73fad8de1018423d81fac

flip2-discrepancies-preparation_code

preparation code · py

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 8a1d691de8fdd9fcf8242fe6fa2cfe2703a1c3281b880fe2a226cb8bd946fd1b

flip2-discrepancies-environment

environment · lock

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 748257abb3664595db1ebf892799e91c6f1a4ca79b0e65eccb3958cf5644a671

Registered recipes and verification receipt

Local recipes: sdk:train-mean-v1; sdk:sequence-composition-v1
Verification: verified at: 2026-09-23T23:00:14.349323+00:00; checks: check: artifact_hashes; status: passed; detail: Exact local bytes recorded; source pins checked where available.; check: join_integrity; status: passed; detail: 184 unique test IDs; scores and outcomes reconcile.; check: score_semantics; status: passed; detail: Higher scores predict higher spectral wavelength; metrics retain their own direction.; check: metric_replay; status: passed; detail: Saved predictions reproduce recorded metrics within 1e-9 absolute tolerance.; check: annotations; status: passed; detail: Registered subgroup fields derive from the preserved source snapshot.; check: dependence; status: passed; detail: Independent sampling unit unknown; descriptive analysis only.; check: recipe_pinned; status: passed; detail: Registered SDK recipe; implementation artifact and environment hashes retained.; check: resource_estimate; status: passed; detail: Local CPU controls previously executed; all jobs remain subject to the campaign watchdog.; limitations: Existing test outcomes are exposed; no independent validation is claimed.; Independent experimental grouping is unavailable; subgroup summaries are descriptive.; Prepared opaque IDs must not be joined to a newly prepared snapshot by ID.; High wavelength is not universally biologically preferable; NDCG is a numerical ranking metric.; Known methodological control, not a novel biological finding.

Read reviewed discrepancy investigations

Evaluation results

1 evaluation · 2 results. Different protocols are not a single leaderboard.

Filter evaluations

Tested model or methodProtocolDatasetEvaluation setupMetricEvidence origin

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configuration	Protocol and dataset	Finding	Evidence and details
Configuration: Training-mean control (FLIP2 Rhomax by_wild_type)	Protocol: FLIP2 Rhomax by_wild_type Dataset subset: FLIP2 Rhomax by_wild_type test subset	0.921 ndcg dimensionless · higher Uncertainty: Not reported Coverage: 184/184	Rewire evaluation · Source checked Methods, coverage and source Training-mean control on FLIP2 Rhomax by_wild_type FLIP2 Rhomax by_wild_type Aggregation: Not reported Training-mean control: local execution report (20 September 2026); Training-mean control: automated execution audit · metrics.ndcg
Configuration: Training-mean control (FLIP2 Rhomax by_wild_type)	Protocol: FLIP2 Rhomax by_wild_type Dataset subset: FLIP2 Rhomax by_wild_type test subset	undefined spearman dimensionless · higher Uncertainty: Not reported Coverage: 184/184	Rewire evaluation · Source checked Methods, coverage and source Training-mean control on FLIP2 Rhomax by_wild_type FLIP2 Rhomax by_wild_type Aggregation: Not reported Training-mean control: local execution report (20 September 2026); Training-mean control: automated execution audit · metrics.spearman

Source checking is not independent reproduction. Release 2026-09-29-06401fd5b220.

Evaluation procedure

FLIP2 Rhomax by_wild_type

Configuration: Training-mean control (FLIP2 Rhomax by_wild_type)
Protocol: FLIP2 Rhomax by_wild_type
Dataset subset: FLIP2 Rhomax by_wild_type test subset

origin: Rewire evaluation
configuration: 1
protocol id: rewire-protocol-flip2-rhomax-by-wild-type-v1
dataset version: rewire-dataset-flip2-rhomax-by-wild-type-v3
split: canonical test
subset: flip2-rhomax-by-wild-type
population: 184/184
inputs: Complete source sequence only; no assay labels during prediction
adaptation: train-only fitting
metric implementation: fcc5ab65e6753cf29aabbdae231cbb5cdf72a3dd40ba8f109802c33223b33536
aggregation: all held-out test rows of the selected target
budget: one local CPU evaluation; no hyperparameter search outside training

Metadata review: source checked. Unreported conditions prevent automatic comparisons.

Reproduction

Split: canonical test
Adaptation: train-only fitting
Scoring implementation: fcc5ab65e6753cf29aabbdae231cbb5cdf72a3dd40ba8f109802c33223b33536

Generate and evaluate predictions

This committed script regenerates the selected local evaluation with the recorded inputs and configuration. Sequence-control instructions execute all four controls; select this evaluation from their outputs. Raw prior predictions are private, so public evidence alone cannot rescore them.

FLIP2 Rhomax by_wild_type: reproduction instructions · Reproduce section

Access requirements and official instructions: FLIP2 Rhomax by_wild_type

Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

132 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-29-06401fd5b220
Property and statement	Original source and location	Review and provenance
attributes.comparison.adaptation train-only fitting Context-only references	FLIP2 Rhomax by_wild_type: reproduction instructions Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.adaptation` Source artifact SHA-256: `75f50703210dc556d77fdfe8f1d28c3200121b5b78c5c6592a096d928b4ff7cc` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.adaptation train-only fitting Context-only references	Training-mean control: local execution report (20 September 2026) Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.adaptation` Source artifact SHA-256: `eb89260f6e138bb9ae2fdf3b882a87fdf8aeabd54a0ac6f17a73f5671c96b4d8` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.aggregation all held-out test rows of the selected target Context-only references	FLIP2 Rhomax by_wild_type: reproduction instructions Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.aggregation` Source artifact SHA-256: `75f50703210dc556d77fdfe8f1d28c3200121b5b78c5c6592a096d928b4ff7cc` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.aggregation all held-out test rows of the selected target Context-only references	Training-mean control: local execution report (20 September 2026) Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.aggregation` Source artifact SHA-256: `eb89260f6e138bb9ae2fdf3b882a87fdf8aeabd54a0ac6f17a73f5671c96b4d8` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.budget one local CPU evaluation; no hyperparameter search outside training Context-only references	FLIP2 Rhomax by_wild_type: reproduction instructions Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.budget` Source artifact SHA-256: `75f50703210dc556d77fdfe8f1d28c3200121b5b78c5c6592a096d928b4ff7cc` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.budget one local CPU evaluation; no hyperparameter search outside training Context-only references	Training-mean control: local execution report (20 September 2026) Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.budget` Source artifact SHA-256: `eb89260f6e138bb9ae2fdf3b882a87fdf8aeabd54a0ac6f17a73f5671c96b4d8` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.dataset_version rewire-dataset-flip2-rhomax-by-wild-type-v3 Context-only references	FLIP2 Rhomax by_wild_type: reproduction instructions Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.dataset_version` Source artifact SHA-256: `75f50703210dc556d77fdfe8f1d28c3200121b5b78c5c6592a096d928b4ff7cc` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.dataset_version rewire-dataset-flip2-rhomax-by-wild-type-v3 Context-only references	Training-mean control: local execution report (20 September 2026) Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.dataset_version` Source artifact SHA-256: `eb89260f6e138bb9ae2fdf3b882a87fdf8aeabd54a0ac6f17a73f5671c96b4d8` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.inputs Complete source sequence only; no assay labels during prediction Context-only references	FLIP2 Rhomax by_wild_type: reproduction instructions Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.inputs` Source artifact SHA-256: `75f50703210dc556d77fdfe8f1d28c3200121b5b78c5c6592a096d928b4ff7cc` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
attributes.comparison.inputs Complete source sequence only; no assay labels during prediction Context-only references	Training-mean control: local execution report (20 September 2026) Original source ↗ metrics; coverage; protocol_results Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: ca73fa47136d182f2d4ddb083d084712198fc0e2 Retrieved: 2026-09-20T20:54:03Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.inputs` Source artifact SHA-256: `eb89260f6e138bb9ae2fdf3b882a87fdf8aeabd54a0ac6f17a73f5671c96b4d8` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

View linked audit checks and correction history

Release 2026-09-29-06401fd5b220 · Record review: source checked

2 source records and release history

Training-mean control: local execution report (20 September 2026) · Original source · ca73fa47136d182f2d4ddb083d084712198fc0e2
FLIP2 Rhomax by_wild_type: reproduction instructions · Original source · ca73fa47136d182f2d4ddb083d084712198fc0e2

Download this release

Technical metadata and extraction receipts

Stable ID: rewire-local-20260920-evaluation-flip2-train-mean

areas: proteins-complexes
origin: rewire_run
execution scope: complete_selected_evaluation
published score reproduction: false
protocol: FLIP2 Rhomax by_wild_type
version: 1
source locator: metrics; coverage; protocol_results
original sdk scope: full
original sdk completion: complete
eligible count: 184
scored count: 184
missing count: 0
coverage: 184/184
suite complete: false
comparison: protocol id: rewire-protocol-flip2-rhomax-by-wild-type-v1; dataset version: rewire-dataset-flip2-rhomax-by-wild-type-v3; split: canonical test; subset: flip2-rhomax-by-wild-type; population: 184/184; inputs: Complete source sequence only; no assay labels during prediction; adaptation: train-only fitting; metric implementation: fcc5ab65e6753cf29aabbdae231cbb5cdf72a3dd40ba8f109802c33223b33536; aggregation: all held-out test rows of the selected target; budget: one local CPU evaluation; no hyperparameter search outside training
input information: Complete source sequence only; no assay labels during prediction
model configuration: constant: arithmetic mean of training targets; fit split: train; validation used: false
provenance: data verification: pinned_source_bytes; dataset version: zenodo-18433203-v3; evaluator url: https://github.com/J-SNACKKB/FLIP/blob/62cace8735f5610e2743cf06ce0f944b37fffaa6/baselines/aggregate.py#L131-L135; source csv sha256: 8e78f6a16cd5298131dca83130d4b94cc0ab4ae9c699460880441c19a65f656f; source sha256: 7c6d2f02cb89310378ac9897c321fbbd909fb0757ac88803dcce84f5f6b05ce3; source url: https://zenodo.org/api/records/18433203/files/rhomax/by_wild_type.csv.gz/content; split origin: Archived assignments: validation=True held out of training; upstream revision: 62cace8735f5610e2743cf06ce0f944b37fffaa6
predictions sha256: 599dca4e6fa643a7cdfc96ac339033f61e5846ce225af17ac2ac3917ecc14fac
prepared sha256: f417d43497e0c8c453bd7c66cfea05959b4d6d47462283096e42a2dae8b72419
execution: adapter: TrainMean; batch size: 32; fitting: train_only; inference and fit seconds: 0.0009417090041097254; mode: local_adapter; prediction type: scalar
timing seconds: evaluation: 0.016455832999781705
environment: architecture: arm64; container digest: unreported; container runtime: unreported; platform: macOS-26.6.2-arm64-arm-64bit; python: 3.11.13; rewirebench: 0.4.0; sdk code sha256: fcc5ab65e6753cf29aabbdae231cbb5cdf72a3dd40ba8f109802c33223b33536; sif sha256: unreported
run url: https://github.com/rewire-bio/rewire-benchmarks/blob/ca73fa47136d182f2d4ddb083d084712198fc0e2/research/local-runs-2026-09-20/flip2-train-mean/report.json
reproduction url: https://github.com/rewire-bio/rewire-benchmarks/blob/ca73fa47136d182f2d4ddb083d084712198fc0e2/research/local-runs-2026-09-20/README.md
limitations: One selected split or assay; no whole-suite score.; No published model score reproduction claim.; Raw predictions retained locally; hashes and inspected contribution bundle are public.; Single execution; no seed variability or uncertainty interval estimated.
reproduction: recipe owner id: rewire-protocol-flip2-rhomax-by-wild-type-v1; recipe id: local-20260920-flip2-repeat; applicability: generate_and_evaluate; explanation: This committed script regenerates the selected local evaluation with the recorded inputs and configuration. Sequence-control instructions execute all four controls; select this evaluation from their outputs. Raw prior predictions are private, so public evidence alone cannot rescore them.; source ids: rewire-local-20260920-instructions-flip2; source locator: Reproduce section

Related records

configuration: Training-mean control (FLIP2 Rhomax by_wild_type)
protocol: FLIP2 Rhomax by_wild_type
dataset subset: FLIP2 Rhomax by_wild_type test subset
evaluation: Training-mean control: ndcg
evaluation: Training-mean control: spearman

Suggest a correction