MFASS v2 evaluates splice-variant prioritisation using a corrected sequence baseline and a fixed grouped holdout.
The eligibility filter category=mutant with a reported strong_lof label selects 27,733 variants and 1,050 disrupting variants from 32,669 source rows. The eligible cohort spans 2,185 exons; the paper’s 2,198-exon count precedes the label filter.
Precision at a fixed review capacity, average precision and AUROC. Each method’s point estimates use its scored subset; paired comparisons use common scored variants.
Sequence baselines use validated 170-base natural_seq reference and original_seq mutant pairs, differing at rel_position minus one with strand-checked alleles. SpliceAI and Pangolin instead use genomic context; their predictions are not evaluations of the same 170-base input.
Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.
limited source coverage · Automated source review, 2026-09-23. All specifications and missing details
Each comparison retains its reviewed evaluation scope, dataset and metric. Results are shown without a pooled ranking.
Precision at 100 (dimensionless) · Higher values are better.
MFASS: matched GENCODE 44 canonical annotation · MFASS v2 test: matched canonical annotation coverage
Evidence origin: Rewire evaluation.
MFASS matched canonical annotation v1: report.json; MFASS matched canonical annotation v1: manifest-v1.json; MFASS matched canonical annotation v1: verification.json; MFASS matched canonical annotation v1: provenance.json; MFASS matched canonical annotation v1: exclusion-verification.json · report.json: conditions.*.metrics.precision_at_capacity8,297 of 8,324 held-out variants scored in every configuration (314 positives, 460 exon groups). The same 27 rows are excluded: 23 assembly-orientation mismatches and four outside the selected canonical transcript spans. Missing scores are not zero or negative predictions.
Automated source review: 2026-09-25. Numerical source review does not establish independent reproduction.
Dots show point estimates. Whiskers show only explicitly defined uncertainty (standard deviation, standard error or a labelled interval); their definitions remain in Table. Unresolved uncertainty is not plotted. Differences do not establish statistical significance.
Showing 4 of 4 matching rows.
Functional exon-recognition labels from the MFASS assay; validated assay-oriented reference/mutant sequence pairs define the sequence-model inputs. The canonical split assigns whole connected exon/gene groups to training or test. Prevalence matching is selected before model runs and uses no predictions. Precision at a fixed review capacity, average precision and AUROC. Each method’s point estimates use its scored subset; paired comparisons use common scored variants. Corrected k-mer/position baseline, frozen DNABERT-2 pair embeddings with a fixed logistic head, and SpliceAI/Pangolin genomic-context specialists. Baseline and logistic-head fitting use training labels only. Exon/gene connected components prevent linked variants crossing arms; exact DNABERT-2 pretraining overlap has not been checked. Paired intervals resample whole connected exon/gene groups; precision intervals account for fixed review-list fraction during group resampling.
Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.
These source-backed links do not make different protocols or scores interchangeable.
Each evaluation records what was tested and under which conditions.
Reference methods help show what a model adds beyond simple controls. We track a null control and a conventional method for each protocol.
2 of 2 active baseline roles have published Rewire measurements in this release. Measurements on a selected protocol do not establish coverage of an entire suite.
3 execution recipes Recipe availability does not establish a completed evaluation.
Literature evidence is not a Rewire measurement. Executed but unpublished runs and private review status are not included.
Published Rewire reference
Training class prior / majority (MFASS v2 canonical held-out split)
Training class prior / majority on MFASS v2 canonical held-out split: methods, coverage and results
Evidence: Reviewed baseline-runs-2026-09-22: MFASS training-prior report metrics, canonical split and coverage; fixed tie-break, not ranking ability
Published Rewire reference
Corrected k-mer / position baseline
Corrected k-mer / position baseline on MFASS v2: methods, coverage and results
Evidence: benchmarks/mfass/results/baseline-kmer-position-v2.json at bee9133b83f3aedaf2bbb9013f1875515845607e; existing release results and corrected split-v2
Protocol coverage CSV · Model evaluation matrix · Source table · Release and checksums
Coverage is derived from release 2026-09-29-06401fd5b220. Source citations describe the original records; they do not validate an unreviewed baseline proposal. No results have been generated by this audit.
Choose a concrete protocol before running an evaluation. Its inputs, split and scoring rules determine which results can be compared.
Evaluate keyed splice-disruption scores on the fixed MFASS v2 test split. Higher scores mean greater splice disruption.
Recompute metrics from supplied predictions. This recipe does not establish reproduction of a particular published score.
Choose one way to run this recipe. These instruction formats are alternatives.
Source reviewed; these instructions have not been executed by rewire.
import rewirebench
prepared = rewirebench.prepare(
"mfass-v2", source="data/mfass", output="prepared"
)
report = rewirebench.evaluate(
prepared, "predictions/predictions.tsv", output="runs/mfass-rescore",
model={"name": "My model", "training_overlap": "Unreported"},
)Run your model locally and return predictions keyed by the input IDs. The evaluator supplies biological inputs without test labels and owns scoring. This interface is not a sandbox for model code.
Pass your existing prediction function into this adapter. Its output direction must match the selected protocol.
class MyModelAdapter:
def __init__(self, score):
self.score = score
def predict(self, inputs):
return {row["id"]: float(self.score(row)) for row in inputs}
# adapter = MyModelAdapter(your_prediction_function)
# report = rewirebench.run(prepared, adapter, output="runs/my-model")Alternatively, generate a keyed prediction file in your existing model environment and use the score-only recipe. Your model code and weights do not need to be shared.
rewirebench: Library guide; rewirebench: MFASS guide; rewirebench: MFASS protocol code; rewirebench: MFASS adapter code · Pinned runner documentation, protocol implementation and adapter implementationContribute a result for review. The library can submit an exported evaluation for private review when intake is open. Check the contribution page for access and sign-in.
Run the corrected MFASS v2 cohort validation and baseline, with an optional DNABERT-2 resource pilot. Keep this protocol separate from the superseded v1 run.
Checked against the official instructions on 2026-09-17. These commands have not been executed by rewire. Running them does not automatically reproduce the published scores.
Repository checkout wrapper: the detached revision selects the exact official source inspected for this guide.
git clone https://github.com/rewire-bio/rewire-benchmarks.git
cd rewire-benchmarks
git checkout --detach 68d1ee53a0fcd1104d9f427d68dedffdccf2c47fOfficial download commands. These upstream master URLs are mutable; the source README records the expected input hashes.
mkdir -p benchmarks/mfass/data
curl -L -o benchmarks/mfass/data/snv_data_clean.txt \
https://raw.githubusercontent.com/KosuriLab/MFASS/master/processed_data/snv/snv_data_clean.txt
curl -L -o benchmarks/mfass/data/snv_func_annot.txt \
https://raw.githubusercontent.com/KosuriLab/MFASS/master/processed_data/snv/snv_func_annot.txtPreserves the official v2 command sequence. The baseline and encoder head are supervised on the predeclared training split.
uv sync --package mfass --extra dnabert2-pilot
uv run --package mfass --extra dnabert2-pilot mfass-build --check
uv run --package mfass --extra dnabert2-pilot mfass-build
uv run --package mfass --extra dnabert2-pilot mfass-baselineThe pilot measures feasibility and validates checkpoint provenance; it does not calculate accuracy. The README gates any subsequent full encoder run on projected runtime and free disk; the full run is intentionally a separate decision.
uv run --package mfass --extra dnabert2-pilot mfass-dnabert2-pilotFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied.
Stable record: rewire-mfass-v2Explanatory profile: limited source coverage · Automated source review, 2026-09-23. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Datasets | The eligibility filter category=mutant with a reported strong_lof label selects 27,733 variants and 1,050 disrupting variants from 32,669 source rows. The eligible cohort spans 2,185 exons; the paper’s 2,198-exon count precedes the label filter.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · README.md lines 40–64 |
| Splits | Canonical split-v2 assigns connected exon/gene components whole: 19,409 training variants (735 positives; 1,127 groups) and 8,324 test variants (315 positives; 463 groups). Assignment uses seed 20260914, 200 candidate draws and a requested 30% test fraction, matching prevalence before predictions. The split TSV SHA-256 is 999ebcb7e63a5c5eaa8780fa468e59ac1f934260ad50102814174c396317f052.Sources (3)MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e; Protocol metadata evidence: mfass-splits-split-v2.manifest.json; Protocol metadata evidence: mfass-splits-split-v2.tsv · README.md lines 66–99; split-v2.manifest.json: seed, restarts, test_frac_requested, train, test; split-v2.tsv bytes |
| Metrics | Precision at a fixed review capacity, average precision and AUROC. Each method’s point estimates use its scored subset; paired comparisons use common scored variants.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Baselines | Corrected k-mer/position baseline, frozen DNABERT-2 pair embeddings with a fixed logistic head, and SpliceAI/Pangolin genomic-context specialists.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Leakage controls | Baseline and logistic-head fitting use training labels only. Exon/gene connected components prevent linked variants crossing arms; exact DNABERT-2 pretraining overlap has not been checked.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Uncertainty | The three pinned candidate-versus-corrected-baseline comparisons use 2,000 paired whole-group bootstrap draws at seed 20260914 and percentile 95% intervals. All report zero single-class draws skipped. The point estimate is the observed candidate-minus-baseline difference on the common subset, not the resample mean. Precision resamples preserve the review-list fraction corresponding to capacity 100, so their realised capacities vary.Sources (5)Protocol metadata evidence: mfass-metrics.py; Protocol metadata evidence: mfass-src-mfass-compare.py; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-spliceai.json; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-pangolin-maskFalse.json; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-dnabert2.json · metrics.py lines 65–140; compare.py lines 33–35, 41–74; comparison JSON: capacity, seed and paired.* |
| Entity type | Paper-specific computational evaluation protocol.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Organisms | Human (Homo sapiens): naturally occurring ExAC variants in or beside human exons, measured in the MFASS minigene assay.SourcesMFASS original assay accession GSE120695 · GSE120695 SOFT: Series_summary and Sample_organism_ch1 fields |
| Assays | MFASS functional exon-recognition labels.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot |
| Allowed inputs | Sequence baselines use validated 170-base natural_seq reference and original_seq mutant pairs, differing at rel_position minus one with strand-checked alleles. SpliceAI and Pangolin instead use genomic context; their predictions are not evaluations of the same 170-base input.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · README.md lines 59–64, 141–179 |
| Adaptation | The corrected k-mer baseline and DNABERT-2 logistic head use the training arm. DNABERT-2 stays frozen: masked-mean last-hidden-state embeddings feed reference and mutant-minus-reference vectors to a fixed balanced L2 logistic head (C=0.1). Specialist scores are zero-shot with respect to MFASS training labels; input context and annotation versions differ.SourcesMFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e · README.md lines 116–138 and 141–179 |
| Scoring coverage | Against the 8,324-variant held-out denominator, the corrected baseline and DNABERT-2 pipeline each score 8,324 variants, SpliceAI scores 8,194, and Pangolin mask=False scores 8,301. Paired comparisons use the common scored variants: 8,194 in 454 groups for SpliceAI, 8,301 in 461 groups for Pangolin, and 8,324 in 463 groups for DNABERT-2. Missing predictions remain outside the paired subset.Sources (3)Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-spliceai.json; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-pangolin-maskFalse.json; Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-dnabert2.json · Each comparison JSON: denominators, independent_groups and on_common_subset; candidate settings retained in filenames |
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Last literature check: 2026-09-17. Dated primary-source discovery and protocol/table screening. Source checking does not mean experimental reproduction. Only separately extracted and independently reviewed numeric batches are publishable.
| Paper or primary resource | Version | Reference |
|---|---|---|
| MFASS benchmark: corrected v2 protocol and reproducibility instructions | Primary full-text snapshot retrieved 2026-09-17; exact bytes pinned by SHA-256 | Read source |
historical results preserved
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
127 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Diagram caption Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-23 Audit detailsFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Diagram steps
| MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-23 Audit detailsFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Computational evaluation flow Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e Pinned README: correction notice; Dataset; Cohort reconciliation; Split; MFASS-v2 results; Paired comparisons; Limits; Archived mfass-v1 snapshot Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-23 Audit detailsFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Datasets The eligibility filter category=mutant with a reported strong_lof label selects 27,733 variants and 1,050 disrupting variants from 32,669 source rows. The eligible cohort spans 2,185 exons; the paper’s 2,198-exon count precedes the label filter. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e README.md lines 40–64 Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-23 Audit detailsFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits Canonical split-v2 assigns connected exon/gene components whole: 19,409 training variants (735 positives; 1,127 groups) and 8,324 test variants (315 positives; 463 groups). Assignment uses seed 20260914, 200 candidate draws and a requested 30% test fraction, matching prevalence before predictions. The split TSV SHA-256 is 999ebcb7e63a5c5eaa8780fa468e59ac1f934260ad50102814174c396317f052. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e README.md lines 66–99; split-v2.manifest.json: seed, restarts, test_frac_requested, train, test; split-v2.tsv bytes Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-23 Audit detailsFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits Canonical split-v2 assigns connected exon/gene components whole: 19,409 training variants (735 positives; 1,127 groups) and 8,324 test variants (315 positives; 463 groups). Assignment uses seed 20260914, 200 candidate draws and a requested 30% test fraction, matching prevalence before predictions. The split TSV SHA-256 is 999ebcb7e63a5c5eaa8780fa468e59ac1f934260ad50102814174c396317f052. Individual claims | Protocol metadata evidence: mfass-splits-split-v2.manifest.json README.md lines 66–99; split-v2.manifest.json: seed, restarts, test_frac_requested, train, test; split-v2.tsv bytes Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-23 Audit detailsFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits Canonical split-v2 assigns connected exon/gene components whole: 19,409 training variants (735 positives; 1,127 groups) and 8,324 test variants (315 positives; 463 groups). Assignment uses seed 20260914, 200 candidate draws and a requested 30% test fraction, matching prevalence before predictions. The split TSV SHA-256 is 999ebcb7e63a5c5eaa8780fa468e59ac1f934260ad50102814174c396317f052. Individual claims | Protocol metadata evidence: mfass-splits-split-v2.tsv README.md lines 66–99; split-v2.manifest.json: seed, restarts, test_frac_requested, train, test; split-v2.tsv bytes Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-23 Audit detailsFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Adaptation The corrected k-mer baseline and DNABERT-2 logistic head use the training arm. DNABERT-2 stays frozen: masked-mean last-hidden-state embeddings feed reference and mutant-minus-reference vectors to a fixed balanced L2 logistic head (C=0.1). Specialist scores are zero-shot with respect to MFASS training labels; input context and annotation versions differ. Individual claims | MFASS computational protocol README at bee9133b83f3aedaf2bbb9013f1875515845607e README.md lines 116–138 and 141–179 Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-23 Audit detailsFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Scoring coverage Against the 8,324-variant held-out denominator, the corrected baseline and DNABERT-2 pipeline each score 8,324 variants, SpliceAI scores 8,194, and Pangolin mask=False scores 8,301. Paired comparisons use the common scored variants: 8,194 in 454 groups for SpliceAI, 8,301 in 461 groups for Pangolin, and 8,324 in 463 groups for DNABERT-2. Missing predictions remain outside the paired subset. Individual claims | Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-dnabert2.json Each comparison JSON: denominators, independent_groups and on_common_subset; candidate settings retained in filenames Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-23 Audit detailsFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Scoring coverage Against the 8,324-variant held-out denominator, the corrected baseline and DNABERT-2 pipeline each score 8,324 variants, SpliceAI scores 8,194, and Pangolin mask=False scores 8,301. Paired comparisons use the common scored variants: 8,194 in 454 groups for SpliceAI, 8,301 in 461 groups for Pangolin, and 8,324 in 463 groups for DNABERT-2. Missing predictions remain outside the paired subset. Individual claims | Protocol metadata evidence: mfass-results-compare-baseline-v2-vs-pangolin-maskFalse.json Each comparison JSON: denominators, independent_groups and on_common_subset; candidate settings retained in filenames Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: bee9133b83f3aedaf2bbb9013f1875515845607e | source checked automated source review · 2026-09-23 Audit detailsFollow-up review of Datasets, Splits, Scoring coverage, Uncertainty, Allowed inputs, Adaptation. Source locations and before/after decisions are recorded in the 23 September profile-evidence audit. Other explanatory content retains its earlier source scope. No human scientific review or independent reproduction is implied. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
View linked audit checks and correction history
Release 2026-09-29-06401fd5b220 · Record review: source checked
Supersedes MFASS v1 (superseded)
Download this releaseStable ID: rewire-mfass-v2