Benchmarking deep learning splice prediction tools using functional splice assays

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 10 methods
• 11 configurations
• 3 protocols
• 3 datasets
• 31 evaluations
• 341 results
• 8 evidence claims
• 1 use case

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• allowed_information: splicing-follow-up-20261009-protocol-riepe2021-abca4-di: Allowed information MMSplice and Spidex could not calculate a score for more than half of the ABCA4 DI variants and were excluded from the DI analysis. (Methods 'In silico splice prediction tools' paragraph 7 (the last))
• Baseline: Prioritise variants for splicing experiments: Baseline MaxEntScan (splicing-follow-up-20261009-source-riepe2021: Introduction (MaxEntScan apart from the classical machine learning tools), Table 1 and Methods (MaxEntScan selection criterion); rna-splicing-20261011-fortier2026-source-article: Introduction (MaxEntScan commonly used))
• CADD: description: Description Combined Annotation-Dependent Depletion: a genome-wide variant deleteriousness score that integrates more than 60 genomic features in a machine learning model trained to separate simulated de novo variants from variants fixed in human populations since the human-chimpanzee split. It scores single nucleotide variants and short insertions and deletions anywhere in the reference assembly, not only missense variants. (Record description; cited Rentzsch et al. 2019, Nucleic Acids Research 47:D886)
• Evidence summary: splicing follow-up experiments: Summary Three independent comparisons score several splice predictors against experimentally measured splicing of human SNVs; none of their authors developed the tools compared. A fourth is reported by a tool's developers. Znabu et al. (2026, preprint) scored all 27,733 labelled MFASS reporter SNVs, of which 1,050 disrupt splicing. Average precision was 0.421 for Pangolin (bootstrap 95% interval 0.386 to 0.454), 0.321 for SpliceAI, 0.317 for SpliceTransformer, 0.256 for MMSplice and 0.228 for SPANR, with AUROC 0.888, 0.819, 0.786, 0.758 and 0.748. SpliceAI was given a score of zero where it returns none. This is a different MFASS population from the matched and v2 held-out protocols and is not pooled with them. Smith and Kitzman (2023) set each tool's threshold so that it flags 10% of 500,000 random exonic and near-exonic SNVs, then measured sensitivity on saturation assays, two of which their own group produced. SpliceAI detected 0.982 of splice-disruptive variants in BRCA1 saturation genome editing, 0.885 in POU1F1 exon 2, 0.860 in WT1 exon 9, 0.539 in FAS exon 6 and 0.237 in MST1R exon 11; Pangolin detected 0.982, 0.948, 0.667, 0.487 and 0.330. Sensitivity depends strongly on the exon, precision is not measured at these thresholds, and HAL and S-Cap score only part of the variants. A curated MLH1 column is recorded but judged outside scope, because its labels include patient RNA and a pathogenicity rule. Riepe et al. (2021) tested up to 11 tools on ABCA4 and MYBPC3 variants assayed in mini- or midigenes. MCC for SpliceAI was 0.42 for ABCA4 noncanonical splice sites, 0.84 for ABCA4 deep-intronic variants and 0.35 for MYBPC3, where the Alamut 3/4 consensus reached 0.53. Those variants were chosen for testing with Alamut or MaxEntScan, and the cutoffs were tuned on the same data. Jaganathan et al. (2026, preprint), the SpliceAI2 developers at Illumina, report Spearman correlation between predicted and measured exon-inclusion change for 283,121 single-nucleotide substitutions in 599 exons of the OpenSplice reporter assay: 0.59 for SpliceAI2, 0.55 for SpliceAI, 0.53 for Pangolin and 0.49 for AlphaGenome (Extended Data Fig. 2). The comparator scores are those released with OpenSplice, while SpliceAI2 was scored by its developers, and a rank correlation over all substitutions is not precision among variants chosen for an experiment. None of these sources measures how many variants a tool would select for a new experiment and how many of those then prove splice-disruptive, so all are proxy evidence for that choice. All values are transcribed from the publications, not reproduced. Reporter and minigene endpoints do not establish patient-RNA effects or pathogenicity, and results from different sources must not be pooled. This summary describes evidence and does not recommend a method. (splicing-follow-up-20261009-source-znabu2026: Section 4 results table; splicing-follow-up-20261009-source-smith2023-table-s2: Table S2, 10% call-rate sheet; splicing-follow-up-20261009-source-smith2023: Methods; splicing-follow-up-20261009-source-riepe2021: Tables 3 to 5; rna-splicing-20261013-spliceai2-source-article: Extended Data Fig. 2 panel 'OpenSplice MPRA (ρ)', Methods 'Variant effect benchmarks')
• Relevance of ABCA4 deep-intronic variants: classification against mini- or midigene splicing results (Riepe et al. Table 4) to "Prioritise variants for splicing experiments": Assessed by ABCA4 deep-intronic variants: classification against mini- or midigene splicing results (Riepe et al. Table 4) (splicing-follow-up-20261009-source-riepe2021: Table 4; Methods 'Datasets' and 'In silico splice prediction tools')
• Relevance of ABCA4 noncanonical splice-site variants: classification against mini- or midigene splicing results (Riepe et al. Table 3) to "Prioritise variants for splicing experiments": Assessed by ABCA4 noncanonical splice-site variants: classification against mini- or midigene splicing results (Riepe et al. Table 3) (splicing-follow-up-20261009-source-riepe2021: Table 3; Methods 'Datasets' and 'In silico splice prediction tools')
• Relevance of MYBPC3 noncanonical splice-site variants: classification against mini- or midigene splicing results (Riepe et al. Table 5) to "Prioritise variants for splicing experiments": Assessed by MYBPC3 noncanonical splice-site variants: classification against mini- or midigene splicing results (Riepe et al. Table 5) (splicing-follow-up-20261009-source-riepe2021: Table 5; Methods 'Datasets' and 'In silico splice prediction tools')
• selection_bias: splicing-follow-up-20261009-protocol-riepe2021-abca4-ncss: Selection bias The selection criterion for functional validation of the ABCA4 variants was a 2% difference in splice score for at least two of the Alamut programs; for MYBPC3 variants, the selection criterion was a lower MaxEntScan score than the score of the reference nucleotide. (Methods 'Datasets' paragraph 2)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.