Evaluation

SpliceAI 1.3.1 on MFASS v2

Research readiness

These checks assess whether the evidence supports a reproducible investigation. A source-checked score alone does not meet these requirements.

Release 2026-09-29-06401fd5b220 · Evidence verified: 2026-09-23

Evidence complete

Replay metrics

Exact outcomes, predictions, identifiers and evaluator are connected.

Supporting evidence

mfass-v2-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
mfass-v2-discrepancies: join_integrity: 8324 unique test IDs; scores and outcomes reconcile.
mfass-v2-discrepancies: score_semantics: Higher scores predict higher assay splice disruption; metrics retain their own direction.
mfass-v2-discrepancies: metric_replay: Saved predictions reproduce recorded metrics within 1e-9 absolute tolerance.

Verified: 2026-09-23

Evidence complete

Investigate discrepancies

Replay evidence includes annotations and an assessment of dependence. Unknown independence permits descriptive analysis only.

Supporting evidence

mfass-v2-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
mfass-v2-discrepancies: join_integrity: 8324 unique test IDs; scores and outcomes reconcile.
mfass-v2-discrepancies: score_semantics: Higher scores predict higher assay splice disruption; metrics retain their own direction.
mfass-v2-discrepancies: metric_replay: Saved predictions reproduce recorded metrics within 1e-9 absolute tolerance.
mfass-v2-discrepancies: annotations: Registered subgroup fields derive from the preserved source snapshot.
mfass-v2-discrepancies: dependence: connected exon/gene group

Verified: 2026-09-23

Evidence incomplete

Run locally

A pinned recipe describes the inputs, environment and resource requirements.

Missing or unresolved evidence

recipe pinned: verification is missing
resource estimate: verification is missing
No pinned local execution recipe is declared.
Local execution requires exactly one pinned prepared artifact.
Local execution requires exactly one pinned recipe code artifact.

Supporting evidence

mfass-v2-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
mfass-v2-discrepancies: join_integrity: 8324 unique test IDs; scores and outcomes reconcile.
mfass-v2-discrepancies: score_semantics: Higher scores predict higher assay splice disruption; metrics retain their own direction.

Verified: 2026-09-23

Evidence incomplete

Validate independently

Separate data and exposure records support an independent test.

Missing or unresolved evidence

independent validation: verification is missing
overlap checked: verification is missing
Data has already been exposed during hypothesis selection; it is not untouched validation.

Supporting evidence

mfass-v2-discrepancies: artifact_hashes: Exact local bytes recorded; source pins checked where available.
mfass-v2-discrepancies: join_integrity: 8324 unique test IDs; scores and outcomes reconcile.
mfass-v2-discrepancies: score_semantics: Higher scores predict higher assay splice disruption; metrics retain their own direction.

Verified: 2026-09-23

Readiness describes the evidence in this release. Availability on your computer is checked separately when an investigation runs. Existing data exposure can prevent independent validation even when files are available.

Limitations

Existing test outcomes have been inspected; all new subgroup findings are exploratory.
Specialists use different genomic context and annotation releases; model-only attribution is unsupported.
Boundary bands are symmetric distances, not canonical dinucleotide annotations; baseline uses distance features.
The historical v1 sequence-orientation error is already corrected and is not a new discovery.
Source assay data have no declared redistribution licence; obtain original tables from KosuriLab/MFASS.
Evidence is exploratory because this data has already been examined.

Artifacts and reproduction

One or more artifacts need a local resolver. Their checksums are recorded, but this release does not provide every download.

MFASS v2: model disagreement

Does model disagreement depend on exon boundary distance, assay replicate agreement or gene/exon concentration?

Manifest: mfass-v2-discrepancies
Prepared outcome: assay splice disruption
Outcome type: binary
Units: binary label
Score direction: higher
Join identifier: id
Unit of independence: connected exon/gene group
Split: test
Prior data exposure: Already used for exploration; not untouched validation

Evaluation protocol and run instructions

Resolve each required artifact from its recorded source, verify its SHA-256, and retain the prepared snapshot and identifiers. A missing public URL means this release does not redistribute that file.

mfass-v2-discrepancies-cohort

cohort · tsv

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 389702ff4c647d7ce10a90092a6fa811ae777d15997baf39ce9aae0346247bd0

mfass-v2-discrepancies-outcomes

outcomes · txt

Artifact source

File SHA-256: a637ca0e307e66ff48811ec7efa22b9ce453bc7883b04f0cacb867f7283132d8

mfass-v2-discrepancies-annotations

annotations · txt

Artifact source

File SHA-256: 71a857fe647c4e68acbb41ca61e959c47e1176de89b1442bd6ca1772aa60d5a1

mfass-v2-discrepancies-split

split · tsv

Artifact source

File SHA-256: 999ebcb7e63a5c5eaa8780fa468e59ac1f934260ad50102814174c396317f052

mfass-v2-discrepancies-baseline-kmer-position-v2-scores

baseline-kmer-position-v2-scores · npy

Artifact source

File SHA-256: ed0ebb3d74183deb1fb475d0e706c0b1211fd6ab9d20449c2c194e57f912d862

mfass-v2-discrepancies-baseline-kmer-position-v2-predictions

baseline-kmer-position-v2-predictions · tsv

Artifact source

File SHA-256: 2f3117c225a8da9ea737abfa2d0f7e1dec97696ae4bacd263864bec9d7f923eb

mfass-v2-discrepancies-baseline-kmer-position-v2-report

baseline-kmer-position-v2-report · json

Artifact source

File SHA-256: 9a0b78674cc714177fec6e8c487588d6186d4fe93d6d7dbcb48bed6858885e15

mfass-v2-discrepancies-dnabert2-117m-frozen-pair-logreg-scores

dnabert2-117m-frozen-pair-logreg-scores · npy

Artifact source

File SHA-256: e6b019babf0b3fb9ec05d4261a9383c0c896183a76c3b31634564c7a63d01105

mfass-v2-discrepancies-dnabert2-117m-frozen-pair-logreg-predictions

dnabert2-117m-frozen-pair-logreg-predictions · tsv

Artifact source

File SHA-256: 3abded2932366e6e2c8d6e0da5836c4240239372758b3c68cd5f19635b8cee11

mfass-v2-discrepancies-dnabert2-117m-frozen-pair-logreg-report

dnabert2-117m-frozen-pair-logreg-report · json

Artifact source

File SHA-256: 60b28541853de349f878d6d1ccbcbe7dfd21b69db976e9c01f60e88bdba0f2a1

mfass-v2-discrepancies-spliceai-1.3.1-predictions

spliceai-1.3.1-predictions · tsv

Artifact source

File SHA-256: b39df773067e4ecd862980f177da69d6117c75c93c754f4cbdd7a77bd419851f

mfass-v2-discrepancies-spliceai-1.3.1-report

spliceai-1.3.1-report · json

Artifact source

File SHA-256: 6d5c59eb0fd60d95064d97e331c9cdb12b2a34db7dc509ff73dfb9502ddf02dd

mfass-v2-discrepancies-pangolin-maskFalse-predictions

pangolin-maskFalse-predictions · tsv

Artifact source

File SHA-256: faa7d4cb3728111f1c5a8e8be6cd2c5958c4cced7c7a366ec96171d03b855ad6

mfass-v2-discrepancies-pangolin-maskFalse-report

pangolin-maskFalse-report · json

Artifact source

File SHA-256: bb0bb6732808699e54938233df1835dfc1f775f33ba7d6acd916e53d588a3c44

mfass-v2-discrepancies-table

table · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: c57f9cfe91d02ab54f6bc20934f5b8f678b37c3b13d8d20aa805fa7d73e62c4f

mfass-v2-discrepancies-receipt

receipt · json

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: a9fa54bcf99457f9b23424ad656a952eb262b376845327e267dc035cf85a012e

mfass-v2-discrepancies-preparation_code

preparation code · py

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 8a1d691de8fdd9fcf8242fe6fa2cfe2703a1c3281b880fe2a226cb8bd946fd1b

mfass-v2-discrepancies-environment

environment · lock

No public download in this release. Obtain access using the documented source and terms.

File SHA-256: 748257abb3664595db1ebf892799e91c6f1a4ca79b0e65eccb3958cf5644a671

Registered recipes and verification receipt

Local recipes: None recorded
Verification: verified at: 2026-09-23T23:00:14.349323+00:00; checks: check: artifact_hashes; status: passed; detail: Exact local bytes recorded; source pins checked where available.; check: join_integrity; status: passed; detail: 8324 unique test IDs; scores and outcomes reconcile.; check: score_semantics; status: passed; detail: Higher scores predict higher assay splice disruption; metrics retain their own direction.; check: metric_replay; status: passed; detail: Saved predictions reproduce recorded metrics within 1e-9 absolute tolerance.; check: annotations; status: passed; detail: Registered subgroup fields derive from the preserved source snapshot.; check: dependence; status: passed; detail: connected exon/gene group; limitations: Existing test outcomes have been inspected; all new subgroup findings are exploratory.; Specialists use different genomic context and annotation releases; model-only attribution is unsupported.; Boundary bands are symmetric distances, not canonical dinucleotide annotations; baseline uses distance features.; The historical v1 sequence-orientation error is already corrected and is not a new discovery.; Source assay data have no declared redistribution licence; obtain original tables from KosuriLab/MFASS.

Read reviewed discrepancy investigations

Evaluation results

1 evaluation · 3 results. Different protocols are not a single leaderboard.

Filter evaluations

Tested model or methodProtocolDatasetEvaluation setupMetricEvidence origin

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configuration	Protocol and dataset	Finding	Evidence and details
Configuration: SpliceAI 1.3.1	Protocol: MFASS v2 Dataset: MFASS v2 eligible assay cohort	0.806 auroc fraction · higher Uncertainty: Not reported Coverage: 8194/8324	Rewire evaluation · Independently reproduced Methods, coverage and source SpliceAI 1.3.1 on MFASS v2 Unchanged specialist run in genomic context with bundled annotation; zero-shot on MFASS assay labels. Point metrics use the scored subset. Aggregation: Not reported MFASS v2 pinned rewire artifacts · benchmarks/mfass/results/spliceai-1.3.1.json :: auroc
Configuration: SpliceAI 1.3.1	Protocol: MFASS v2 Dataset: MFASS v2 eligible assay cohort	0.299 average_precision fraction · higher Uncertainty: Not reported Coverage: 8194/8324	Rewire evaluation · Independently reproduced Methods, coverage and source SpliceAI 1.3.1 on MFASS v2 Unchanged specialist run in genomic context with bundled annotation; zero-shot on MFASS assay labels. Point metrics use the scored subset. Aggregation: Not reported MFASS v2 pinned rewire artifacts · benchmarks/mfass/results/spliceai-1.3.1.json :: average_precision
Configuration: SpliceAI 1.3.1	Protocol: MFASS v2 Dataset: MFASS v2 eligible assay cohort	0.64 precision_at_100 fraction · higher Uncertainty: Not reported Coverage: 8194/8324	Rewire evaluation · Independently reproduced Methods, coverage and source SpliceAI 1.3.1 on MFASS v2 Unchanged specialist run in genomic context with bundled annotation; zero-shot on MFASS assay labels. Point metrics use the scored subset. Aggregation: Not reported MFASS v2 pinned rewire artifacts · benchmarks/mfass/results/spliceai-1.3.1.json :: precision_at_100

Source checking is not independent reproduction. Release 2026-09-29-06401fd5b220.

Evaluation procedure

Unchanged specialist run in genomic context with bundled annotation; zero-shot on MFASS assay labels. Point metrics use the scored subset.

Configuration: SpliceAI 1.3.1
Protocol: MFASS v2
Dataset: MFASS v2 eligible assay cohort

origin: Rewire evaluation
configuration: bee9133b83f3aedaf2bbb9013f1875515845607e
protocol id: mfass-v2
dataset version: bee9133b83f3aedaf2bbb9013f1875515845607e
split: split-v2.tsv
population: 8194/8324
inputs: Not reported
adaptation: Not reported
metric implementation: Not reported
aggregation: Not reported
budget: Not reported

Metadata review: reproduced. Unreported conditions prevent automatic comparisons.

Reproduction

Split: split-v2.tsv
Adaptation: Not reported
Scoring implementation: Not reported

Recompute metrics from existing predictions

This mapping supports rescoring the original held-out prediction IDs under the corrected MFASS v2 protocol. It does not regenerate model predictions or establish reproduction of the published score. Preserve the reported missing-prediction coverage.

rewirebench: MFASS guide; rewirebench: MFASS protocol code · docs/mfass.md: archived predictions; protocols/mfass.py: canonical split, score and coverage

Access requirements and official instructions: MFASS v2

Reproducing this published result requires matching its model configuration, data, split and scorer. Source checking or a successful smoke test does not establish score reproduction.

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

75 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-29-06401fd5b220
Property and statement	Original source and location	Review and provenance
attributes.comparison.adaptation Not reported Context-only references	MFASS v2 pinned rewire artifacts Original source ↗ No field-specific location recorded Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bee9133b83f3aedaf2bbb9013f1875515845607e Retrieved: 2026-09-16T10:50:02Z	missing or unspecified No individual claim review recorded rewire run Audit details Field: `attributes.comparison.adaptation` Source artifact SHA-256: `a3af693afc070b39b334e359beed7f9affd76f7456a4f9264e2d1e9dab26111d` Hash scope: Local imported data/benchmark-runs/mfass-v2.json; not the upstream repository or result-file hash
attributes.comparison.adaptation Not reported Context-only references	rewirebench: MFASS guide Original source ↗ No field-specific location recorded Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87 Retrieved: 2026-09-17	missing or unspecified No individual claim review recorded rewire run Audit details Field: `attributes.comparison.adaptation` Source artifact SHA-256: `53f2d58c152be488bf1c7a322a4a09574e0c87192e9eac19aa1f780f8eb6559c` Hash scope: complete file bytes Format: text
attributes.comparison.adaptation Not reported Context-only references	rewirebench: MFASS protocol code Original source ↗ No field-specific location recorded Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87 Retrieved: 2026-09-17	missing or unspecified No individual claim review recorded rewire run Audit details Field: `attributes.comparison.adaptation` Source artifact SHA-256: `6d090a6b1622fe571b8c48ecc213b9b6adf3cbf06cc2abc2d3bfc891a3f8ed93` Hash scope: complete file bytes Format: text
attributes.comparison.aggregation Not reported Context-only references	MFASS v2 pinned rewire artifacts Original source ↗ No field-specific location recorded Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bee9133b83f3aedaf2bbb9013f1875515845607e Retrieved: 2026-09-16T10:50:02Z	missing or unspecified No individual claim review recorded rewire run Audit details Field: `attributes.comparison.aggregation` Source artifact SHA-256: `a3af693afc070b39b334e359beed7f9affd76f7456a4f9264e2d1e9dab26111d` Hash scope: Local imported data/benchmark-runs/mfass-v2.json; not the upstream repository or result-file hash
attributes.comparison.aggregation Not reported Context-only references	rewirebench: MFASS guide Original source ↗ No field-specific location recorded Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87 Retrieved: 2026-09-17	missing or unspecified No individual claim review recorded rewire run Audit details Field: `attributes.comparison.aggregation` Source artifact SHA-256: `53f2d58c152be488bf1c7a322a4a09574e0c87192e9eac19aa1f780f8eb6559c` Hash scope: complete file bytes Format: text
attributes.comparison.aggregation Not reported Context-only references	rewirebench: MFASS protocol code Original source ↗ No field-specific location recorded Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87 Retrieved: 2026-09-17	missing or unspecified No individual claim review recorded rewire run Audit details Field: `attributes.comparison.aggregation` Source artifact SHA-256: `6d090a6b1622fe571b8c48ecc213b9b6adf3cbf06cc2abc2d3bfc891a3f8ed93` Hash scope: complete file bytes Format: text
attributes.comparison.budget Not reported Context-only references	MFASS v2 pinned rewire artifacts Original source ↗ No field-specific location recorded Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bee9133b83f3aedaf2bbb9013f1875515845607e Retrieved: 2026-09-16T10:50:02Z	missing or unspecified No individual claim review recorded rewire run Audit details Field: `attributes.comparison.budget` Source artifact SHA-256: `a3af693afc070b39b334e359beed7f9affd76f7456a4f9264e2d1e9dab26111d` Hash scope: Local imported data/benchmark-runs/mfass-v2.json; not the upstream repository or result-file hash
attributes.comparison.budget Not reported Context-only references	rewirebench: MFASS guide Original source ↗ No field-specific location recorded Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87 Retrieved: 2026-09-17	missing or unspecified No individual claim review recorded rewire run Audit details Field: `attributes.comparison.budget` Source artifact SHA-256: `53f2d58c152be488bf1c7a322a4a09574e0c87192e9eac19aa1f780f8eb6559c` Hash scope: complete file bytes Format: text
attributes.comparison.budget Not reported Context-only references	rewirebench: MFASS protocol code Original source ↗ No field-specific location recorded Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: f80cef7f818bec33e51b7f43ad499eb5078c8d87 Retrieved: 2026-09-17	missing or unspecified No individual claim review recorded rewire run Audit details Field: `attributes.comparison.budget` Source artifact SHA-256: `6d090a6b1622fe571b8c48ecc213b9b6adf3cbf06cc2abc2d3bfc891a3f8ed93` Hash scope: complete file bytes Format: text
attributes.comparison.dataset_version bee9133b83f3aedaf2bbb9013f1875515845607e Context-only references	MFASS v2 pinned rewire artifacts Original source ↗ No field-specific location recorded Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bee9133b83f3aedaf2bbb9013f1875515845607e Retrieved: 2026-09-16T10:50:02Z	not individually reviewed No individual claim review recorded rewire run Audit details Field: `attributes.comparison.dataset_version` Source artifact SHA-256: `a3af693afc070b39b334e359beed7f9affd76f7456a4f9264e2d1e9dab26111d` Hash scope: Local imported data/benchmark-runs/mfass-v2.json; not the upstream repository or result-file hash

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

View linked audit checks and correction history

Release 2026-09-29-06401fd5b220 · Record review: reproduced

3 source records and release history

MFASS v2 pinned rewire artifacts · Original source · bee9133b83f3aedaf2bbb9013f1875515845607e
rewirebench: MFASS guide · Original source · f80cef7f818bec33e51b7f43ad499eb5078c8d87
rewirebench: MFASS protocol code · Original source · f80cef7f818bec33e51b7f43ad499eb5078c8d87

Download this release

Technical metadata and extraction receipts

Stable ID: rewire-evaluation-spliceai-1-3-1

areas: dna-genomes
origin: rewire_run
protocol: Unchanged specialist run in genomic context with bundled annotation; zero-shot on MFASS assay labels. Point metrics use the scored subset.
version: bee9133b83f3aedaf2bbb9013f1875515845607e
run url: /benchmarks/runs/mfass-v2/
comparison: protocol id: mfass-v2; dataset version: bee9133b83f3aedaf2bbb9013f1875515845607e; split: split-v2.tsv; population: 8194/8324; inputs: Not reported; adaptation: Not reported; metric implementation: Not reported; aggregation: Not reported; budget: Not reported
coverage: 8194/8324
missing metadata: paired comparison: Refer to paired bootstrap artifacts; marginal scores are not paired comparisons.
reproduction: recipe owner id: rewire-mfass-v2; recipe id: mfass-v2-rescore; applicability: rescore_predictions; explanation: This mapping supports rescoring the original held-out prediction IDs under the corrected MFASS v2 protocol. It does not regenerate model predictions or establish reproduction of the published score. Preserve the reported missing-prediction coverage.; source ids: runner-recipe-docs-mfass-md-f80cef7f; runner-recipe-packages-rewirebench-src-rewirebench-protocols-mfass-py-f80cef7f; source locator: docs/mfass.md: archived predictions; protocols/mfass.py: canonical split, score and coverage

Related records

model: SpliceAI 1.3.1
benchmark: MFASS v2
dataset: MFASS v2 eligible assay cohort
evaluation: SpliceAI 1.3.1 · auroc
evaluation: SpliceAI 1.3.1 · average_precision
evaluation: SpliceAI 1.3.1 · precision_at_100

Suggest a correction