rewire.itbenchmarks
Task

MFASS splice-variant prioritisation

MFASS prioritisation asks whether variant scores enrich for experimentally disrupted exon recognition.

SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files

9 evaluations · 31 results

Overview

Datasets

The linked rewire MFASS v2 protocol defines its reconciled eligible variants and fixed test cohort; this guide is not another dataset release.

Metrics

Precision at 100, average precision and AUROC in the pinned v2 evaluation. Coverage and paired uncertainty belong beside each comparison.

Allowed inputs

Variant scores aligned to the MFASS reporter-assay labels; assay-oriented sequence pairs for the corrected local baseline.

SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
Evaluation procedure diagram
How it worksConceptual evaluation workflow
Conceptual evaluation workflow1. Validate assay-oriented sequence pairs. Then: 2. Apply the fixed grouped split. Then: 3. Score each declared method. Then: 4. Report prioritisation, ranking and coverageConceptual evaluation workflow1. Validate assay-oriented sequence pairs. Then: 2. Apply the fixed grouped split. Then: 3. Score each declared method. Then: 4. Report prioritisation, ranking and coverageConceptual evaluation workflow1. Validate assay-oriented sequence pairs. Then: 2. Apply the fixed grouped split. Then: 3. Score each declared method. Then: 4. Report prioritisation, ranking and coverage

Conceptual task guide. Dataset preparation, parameters and scoring must come from a separately identified protocol.

SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files

limited source coverage · Automated source review, 2026-09-23. All specifications and missing details

Results

Each comparison retains its reviewed evaluation scope, dataset and metric. Results are shown without a pooled ranking.

MFASS matched canonical annotation: Precision at 100

Precision at 100 (dimensionless) · Higher values are better.

MFASS: matched GENCODE 44 canonical annotation · MFASS v2 test: matched canonical annotation coverage

Evidence origin: Rewire evaluation.

MFASS matched canonical annotation v1: report.json; MFASS matched canonical annotation v1: manifest-v1.json; MFASS matched canonical annotation v1: verification.json; MFASS matched canonical annotation v1: provenance.json; MFASS matched canonical annotation v1: exclusion-verification.json · report.json: conditions.*.metrics.precision_at_capacity
  • 8,297 of 8,324 held-out variants scored in every configuration (314 positives, 460 exon groups). The same 27 rows are excluded: 23 assembly-orientation mismatches and four outside the selected canonical transcript spans. Missing scores are not zero or negative predictions.
  • Exploratory comparison: prior results were known. Paired contrast intervals are unadjusted and do not establish a universal model ranking.
  • Pangolin uses the recorded per-gene masking patch; these are exact configurations, not unqualified upstream model scores.
  • Precision at 100 is sensitive to tied-score ordering, especially P1. Numerical source checking is automated, not human review or independent reproduction.
  • Assembly-orientation issue reported at https://github.com/KosuriLab/MFASS/issues/1. Original v1 outputs remain unchanged; corrections require a new version.
Comparison details and limitations

8,297 of 8,324 held-out variants scored in every configuration (314 positives, 460 exon groups). The same 27 rows are excluded: 23 assembly-orientation mismatches and four outside the selected canonical transcript spans. Missing scores are not zero or negative predictions.

Automated source review: 2026-09-25. Numerical source review does not establish independent reproduction.

Dots show point estimates. Whiskers show only explicitly defined uncertainty (standard deviation, standard error or a labelled interval); their definitions remain in Table. Unresolved uncertainty is not plotted. Differences do not establish statistical significance.

Showing 4 of 4 matching rows.

Methods and evaluation design

Procedure, tasks and evaluated configurations

How it works

Choosing an evaluation

The corrected rewire v2 protocol validates assay-oriented reference and mutant pairs and uses a fixed grouped split. Its baseline window is placed around the validated variant position. This task guide links that concrete protocol without replacing its identity. Preserve specialist sequence context, missing predictions and the chosen review capacity when comparing methods.

SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
Task scope

This is a task guide, not a single versioned benchmark protocol. The connected resources provide examples or concrete procedures. A candidate method or proposed control is not evidence that an evaluation has been completed.

SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files

Evaluation design

Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.

These source-backed links do not make different protocols or scores interchangeable.

Run this benchmark

Choose a concrete protocol before running an evaluation. Its inputs, split and scoring rules determine which results can be compared.

Run instructions

No runnable recipe has been reviewed for this task. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.

A task describes a biological question. Choose a linked protocol to obtain concrete split and scoring instructions.

Strengths, limitations and unresolved questions

Strengths and limitations

Strengths and considerations

  • Functional reporter labels provide an assay endpoint separate from clinical assertions.
    SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files

Limitations and conditions

  • Reporter effects are not clinical diagnoses. Different sequence context and missing-prediction coverage prevent an identical-input interpretation.
    SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
  • The archived v1 baseline is superseded for an orientation/window error. Its preserved comparisons and split-cost claims are not current evidence for the corrected v2 protocol.
    SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · README.md correction notice lines 3–10
Profile review details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Stable record: catalog-task-mfass-splice

Specifications

Inputs, training, access and other details

Explanatory profile: limited source coverage · Automated source review, 2026-09-23. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
PropertyDescription and evidence
Entity typeTask guide; concrete protocol identities remain separate.
SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
DatasetsThe linked rewire MFASS v2 protocol defines its reconciled eligible variants and fixed test cohort; this guide is not another dataset release.
SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
OrganismsHuman variants evaluated through the MFASS reporter assay; this is not a population-level clinical validation.
SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
AssaysMFASS reporter-based exon-recognition measurements, using the labels and eligibility rules preserved by the v2 protocol.
SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
SplitsUse the grouped split-v2 manifests in the pinned runner revision. Preserve the train/test assignment and exclusions.
SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
Allowed inputsVariant scores aligned to the MFASS reporter-assay labels; assay-oriented sequence pairs for the corrected local baseline.
SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
AdaptationThe corrected baseline and frozen-encoder logistic pipeline fit the training arm; the specialist scorers retain their published configurations.
SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
MetricsPrecision at 100, average precision and AUROC in the pinned v2 evaluation. Coverage and paired uncertainty belong beside each comparison.
SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
BaselinesThe v2 k-mer/position baseline is a trained comparator. SpliceAI, Pangolin and the frozen DNABERT-2 logistic pipeline retain their own input and fitting definitions.
SourcesMFASS v2 pinned rewire artifacts · Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files
Coverage and comparison scopeThe linked rewire v2 protocol has 8,324 held-out variants, but scored coverage depends on the method. Compare candidate and baseline on their common scored variants and retain excluded counts and connected-group counts. This guide defines no additional cohort or split.
Sources (4)MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-spliceai.json; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-pangolin-maskFalse.json; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-dnabert2.json · README.md lines 141–171; comparison JSON: denominators and independent_groups
UncertaintyFor the linked v2 comparisons, intervals describe paired candidate-minus-baseline differences from 2,000 whole-group bootstrap draws. They do not represent absolute method confidence intervals; capacity-100 precision keeps the review-list fraction fixed during resampling.
Sources (4)Protocol metadata evidence: mfass-metrics.py; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-spliceai.json; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-pangolin-maskFalse.json; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-dnabert2.json · metrics.py lines 65–140; comparison JSON: capacity, seed and paired.*

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Papers and result coverage

Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.

Paper or primary resourceVersionReference
mfass originalPMC primary-source artifact, version pinned by SHA256Read source
A Multiplexed Assay for Exon Recognition Reveals that an Unappreciated Fraction of Rare Genetic Variants Cause Large-Effect Splicing DisruptionsOriginal published articleRead source
DOI: 10.1016/j.molcel.2018.10.037
Historical gaps recorded on 2026-09-17

The catalogue now holds 31 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.

  • Primary browser text accessible but rawXML endpoint 500, PMC HTMLreCAPTCHA and mirror returned 0 bytes. Existing MFASS corrected runs must retain identities and assay-oriented window correction; no new score duplicates or revert to withdrawn v 1 conclusion.
Search and extraction details

broad task primary raw retrieval blocked existing corrected run retained

Searches

  • MFASS v2 pinned rewire artifacts primary paper benchmark results

Evidence locations

  • Original MFASS paper discovered through PMC/DOI; assay design and variant library; existing Rewire v2 artifacts

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

26 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-29-06401fd5b220
Property and statementOriginal source and locationReview and provenance
Diagram caption
Conceptual task guide. Dataset preparation, parameters and scoring must come from a separately identified protocol.
Individual claims
MFASS v2 pinned rewire artifacts

Original source ↗

Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files

Version: bee9133b83f3aedaf2bbb9013f1875515845607e
Retrieved: 2026-09-16T10:50:02Z

source checked

automated source review · 2026-09-23

Audit details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Field: attributes.profile.diagram.caption

Source artifact SHA-256: a3af693afc070b39b334e359beed7f9affd76f7456a4f9264e2d1e9dab26111d

Hash scope: Local imported data/benchmark-runs/mfass-v2.json; not the upstream repository or result-file hash

Diagram steps
  • Validate assay-oriented sequence pairs
  • Apply the fixed grouped split
  • Score each declared method
  • Report prioritisation, ranking and coverage
Individual claims
MFASS v2 pinned rewire artifacts

Original source ↗

Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files

Version: bee9133b83f3aedaf2bbb9013f1875515845607e
Retrieved: 2026-09-16T10:50:02Z

source checked

automated source review · 2026-09-23

Audit details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Field: attributes.profile.diagram.steps

Source artifact SHA-256: a3af693afc070b39b334e359beed7f9affd76f7456a4f9264e2d1e9dab26111d

Hash scope: Local imported data/benchmark-runs/mfass-v2.json; not the upstream repository or result-file hash

Diagram title
Conceptual evaluation workflow
Individual claims
MFASS v2 pinned rewire artifacts

Original source ↗

Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files

Version: bee9133b83f3aedaf2bbb9013f1875515845607e
Retrieved: 2026-09-16T10:50:02Z

source checked

automated source review · 2026-09-23

Audit details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Field: attributes.profile.diagram.title

Source artifact SHA-256: a3af693afc070b39b334e359beed7f9affd76f7456a4f9264e2d1e9dab26111d

Hash scope: Local imported data/benchmark-runs/mfass-v2.json; not the upstream repository or result-file hash

Entity type
Task guide; concrete protocol identities remain separate.
Individual claims
MFASS v2 pinned rewire artifacts

Original source ↗

Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files

Version: bee9133b83f3aedaf2bbb9013f1875515845607e
Retrieved: 2026-09-16T10:50:02Z

source checked

automated source review · 2026-09-23

Audit details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Field: attributes.profile.facts.0.value

Source artifact SHA-256: a3af693afc070b39b334e359beed7f9affd76f7456a4f9264e2d1e9dab26111d

Hash scope: Local imported data/benchmark-runs/mfass-v2.json; not the upstream repository or result-file hash

Datasets
The linked rewire MFASS v2 protocol defines its reconciled eligible variants and fixed test cohort; this guide is not another dataset release.
Individual claims
MFASS v2 pinned rewire artifacts

Original source ↗

Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files

Version: bee9133b83f3aedaf2bbb9013f1875515845607e
Retrieved: 2026-09-16T10:50:02Z

source checked

automated source review · 2026-09-23

Audit details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Field: attributes.profile.facts.1.value

Source artifact SHA-256: a3af693afc070b39b334e359beed7f9affd76f7456a4f9264e2d1e9dab26111d

Hash scope: Local imported data/benchmark-runs/mfass-v2.json; not the upstream repository or result-file hash

Uncertainty
For the linked v2 comparisons, intervals describe paired candidate-minus-baseline differences from 2,000 whole-group bootstrap draws. They do not represent absolute method confidence intervals; capacity-100 precision keeps the review-list fraction fixed during resampling.
Individual claims
Protocol metadata evidence: mfass-metrics.py

Original source ↗

metrics.py lines 65–140; comparison JSON: capacity, seed and paired.*

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: bee9133b83f3aedaf2bbb9013f1875515845607e
Retrieved: 2026-09-23T18:43:09.582943+00:00

source checked

automated source review · 2026-09-23

Audit details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Field: attributes.profile.facts.10.value

Source artifact SHA-256: 4030374ed11d38e523fc9991ddd792f8800e7739ac71588085bfff91f912d7f7

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Uncertainty
For the linked v2 comparisons, intervals describe paired candidate-minus-baseline differences from 2,000 whole-group bootstrap draws. They do not represent absolute method confidence intervals; capacity-100 precision keeps the review-list fraction fixed during resampling.
Individual claims
Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-dnabert2.json

Original source ↗

metrics.py lines 65–140; comparison JSON: capacity, seed and paired.*

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: bee9133b83f3aedaf2bbb9013f1875515845607e
Retrieved: 2026-09-23T18:39:03.239354+00:00

source checked

automated source review · 2026-09-23

Audit details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Field: attributes.profile.facts.10.value

Source artifact SHA-256: 52216e003f4e5332ecb43511864c9f0b3df7b1ed0f731c631412dc7311f505c3

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Uncertainty
For the linked v2 comparisons, intervals describe paired candidate-minus-baseline differences from 2,000 whole-group bootstrap draws. They do not represent absolute method confidence intervals; capacity-100 precision keeps the review-list fraction fixed during resampling.
Individual claims
Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-pangolin-maskFalse.json

Original source ↗

metrics.py lines 65–140; comparison JSON: capacity, seed and paired.*

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: bee9133b83f3aedaf2bbb9013f1875515845607e
Retrieved: 2026-09-23T18:39:03.230617+00:00

source checked

automated source review · 2026-09-23

Audit details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Field: attributes.profile.facts.10.value

Source artifact SHA-256: fd938e1414fa1c6611c6b62597c7b7a3c2df0078301e47e23ba82183095c1097

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Uncertainty
For the linked v2 comparisons, intervals describe paired candidate-minus-baseline differences from 2,000 whole-group bootstrap draws. They do not represent absolute method confidence intervals; capacity-100 precision keeps the review-list fraction fixed during resampling.
Individual claims
Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-spliceai.json

Original source ↗

metrics.py lines 65–140; comparison JSON: capacity, seed and paired.*

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: bee9133b83f3aedaf2bbb9013f1875515845607e
Retrieved: 2026-09-23T18:39:03.206346+00:00

source checked

automated source review · 2026-09-23

Audit details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Field: attributes.profile.facts.10.value

Source artifact SHA-256: 7790098265ad4cac961e26d44ecd197fc314f5214891fc93d452d2eca65f62e0

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Organisms
Human variants evaluated through the MFASS reporter assay; this is not a population-level clinical validation.
Individual claims
MFASS v2 pinned rewire artifacts

Original source ↗

Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files

Version: bee9133b83f3aedaf2bbb9013f1875515845607e
Retrieved: 2026-09-16T10:50:02Z

source checked

automated source review · 2026-09-23

Audit details

Follow-up review of Coverage and comparison scope, Uncertainty. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.

Field: attributes.profile.facts.2.value

Source artifact SHA-256: a3af693afc070b39b334e359beed7f9affd76f7456a4f9264e2d1e9dab26111d

Hash scope: Local imported data/benchmark-runs/mfass-v2.json; not the upstream repository or result-file hash

Sources and history

View linked audit checks and correction history

Release 2026-09-29-06401fd5b220 · Record review: discovered

12 source records and release historyDownload this release
Technical metadata and extraction receipts

Stable ID: catalog-task-mfass-splice

areas
dna-genomes
entity level
task
version
Not reported
task
MFASS splice-variant prioritisation
scope note
Functional exon-recognition assay; mfass-v2 reports a corrected baseline and one complete local DNABERT-2 protocol.
benchmark research
review date: 2026-09-17; status: broad_task_primary_raw_retrieval_blocked_existing_corrected_run_retained; primary sources: evidence-expansion-p2-mfass-original-noartifact-2; evidence-expansion-p2-mfass-primary-bibliographic-noartifact-2; inspected locators: Original MFASS paper discovered through PMC/DOI; assay design and variant library; existing Rewire v2 artifacts; searched queries: MFASS v2 pinned rewire artifacts primary paper benchmark results; gaps: Primary browser text accessible but rawXML endpoint 500, PMC HTMLreCAPTCHA and mirror returned 0 bytes. Existing MFASS corrected runs must retain identities and assay-oriented window correction; no new score duplicates or revert to withdrawn v 1 conclusion.; claim scope: Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
historical missing metadata
protocol version: not_yet_extracted
metadata review scope
historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
legacy kinds
benchmark
entity classification
review date: 2026-09-17; rationale: This record identifies the biological prediction question or a suite-specific task, rather than a uniquely fixed evaluated procedure. Preserve its task identity and leave split, model adaptation and scoring details on linked protocols/evaluations.; source ids: rewire-mfass-v2-source; source locator: Pinned MFASS v2 README: Correction, Dataset, Cohort reconciliation and Split; committed results files; ambiguities: None recorded
Related records

Suggest a correction