A Reproducible MFASS Benchmark of Splice-Disruption Predictors Reveals a Shared Exon-Interior Blind Spot

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 3 methods
• 5 configurations
• 1 protocol
• 5 evaluations
• 15 results
• 3 evidence claims
• 1 use case

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• Evidence summary: splicing follow-up experiments: Summary Three independent comparisons score several splice predictors against experimentally measured splicing of human SNVs; none of their authors developed the tools compared. A fourth is reported by a tool's developers. Znabu et al. (2026, preprint) scored all 27,733 labelled MFASS reporter SNVs, of which 1,050 disrupt splicing. Average precision was 0.421 for Pangolin (bootstrap 95% interval 0.386 to 0.454), 0.321 for SpliceAI, 0.317 for SpliceTransformer, 0.256 for MMSplice and 0.228 for SPANR, with AUROC 0.888, 0.819, 0.786, 0.758 and 0.748. SpliceAI was given a score of zero where it returns none. This is a different MFASS population from the matched and v2 held-out protocols and is not pooled with them. Smith and Kitzman (2023) set each tool's threshold so that it flags 10% of 500,000 random exonic and near-exonic SNVs, then measured sensitivity on saturation assays, two of which their own group produced. SpliceAI detected 0.982 of splice-disruptive variants in BRCA1 saturation genome editing, 0.885 in POU1F1 exon 2, 0.860 in WT1 exon 9, 0.539 in FAS exon 6 and 0.237 in MST1R exon 11; Pangolin detected 0.982, 0.948, 0.667, 0.487 and 0.330. Sensitivity depends strongly on the exon, precision is not measured at these thresholds, and HAL and S-Cap score only part of the variants. A curated MLH1 column is recorded but judged outside scope, because its labels include patient RNA and a pathogenicity rule. Riepe et al. (2021) tested up to 11 tools on ABCA4 and MYBPC3 variants assayed in mini- or midigenes. MCC for SpliceAI was 0.42 for ABCA4 noncanonical splice sites, 0.84 for ABCA4 deep-intronic variants and 0.35 for MYBPC3, where the Alamut 3/4 consensus reached 0.53. Those variants were chosen for testing with Alamut or MaxEntScan, and the cutoffs were tuned on the same data. Jaganathan et al. (2026, preprint), the SpliceAI2 developers at Illumina, report Spearman correlation between predicted and measured exon-inclusion change for 283,121 single-nucleotide substitutions in 599 exons of the OpenSplice reporter assay: 0.59 for SpliceAI2, 0.55 for SpliceAI, 0.53 for Pangolin and 0.49 for AlphaGenome (Extended Data Fig. 2). The comparator scores are those released with OpenSplice, while SpliceAI2 was scored by its developers, and a rank correlation over all substitutions is not precision among variants chosen for an experiment. None of these sources measures how many variants a tool would select for a new experiment and how many of those then prove splice-disruptive, so all are proxy evidence for that choice. All values are transcribed from the publications, not reproduced. Reporter and minigene endpoints do not establish patient-RNA effects or pathogenicity, and results from different sources must not be pooled. This summary describes evidence and does not recommend a method. (splicing-follow-up-20261009-source-znabu2026: Section 4 results table; splicing-follow-up-20261009-source-smith2023-table-s2: Table S2, 10% call-rate sheet; splicing-follow-up-20261009-source-smith2023: Methods; splicing-follow-up-20261009-source-riepe2021: Tables 3 to 5; rna-splicing-20261013-spliceai2-source-article: Extended Data Fig. 2 panel 'OpenSplice MPRA (ρ)', Methods 'Variant effect benchmarks')
• Relevance of MFASS splice-disrupting variant detection, all 27,733 labelled SNVs (Znabu et al. 2026) to "Prioritise variants for splicing experiments": Assessed by MFASS splice-disrupting variant detection, all 27,733 labelled SNVs (Znabu et al. 2026) (splicing-follow-up-20261009-source-znabu2026: Section 4 Results table; Sections 2 and 3)
• reported_finding: splicing-follow-up-20261009-protocol-znabu2026-mfass-full-cohort: Reported finding 200 of 1,050 disrupting variants (19%) are missed by all five tools at a common 10% false-positive operating point; 77% of these lie more than 10 bp from a splice site and 149 of 200 are exonic. (Section 5 'Where the tools fail' paragraph 3)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.