Benchmarking splice variant prediction algorithms using massively parallel splicing assays

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 5 methods
• 8 configurations
• 6 protocols
• 6 datasets
• 48 evaluations
• 166 results
• 8 evidence claims
• 1 use case

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• Evidence summary: splicing follow-up experiments: Summary Three independent comparisons score several splice predictors against experimentally measured splicing of human SNVs; none of their authors developed the tools compared. A fourth is reported by a tool's developers. Znabu et al. (2026, preprint) scored all 27,733 labelled MFASS reporter SNVs, of which 1,050 disrupt splicing. Average precision was 0.421 for Pangolin (bootstrap 95% interval 0.386 to 0.454), 0.321 for SpliceAI, 0.317 for SpliceTransformer, 0.256 for MMSplice and 0.228 for SPANR, with AUROC 0.888, 0.819, 0.786, 0.758 and 0.748. SpliceAI was given a score of zero where it returns none. This is a different MFASS population from the matched and v2 held-out protocols and is not pooled with them. Smith and Kitzman (2023) set each tool's threshold so that it flags 10% of 500,000 random exonic and near-exonic SNVs, then measured sensitivity on saturation assays, two of which their own group produced. SpliceAI detected 0.982 of splice-disruptive variants in BRCA1 saturation genome editing, 0.885 in POU1F1 exon 2, 0.860 in WT1 exon 9, 0.539 in FAS exon 6 and 0.237 in MST1R exon 11; Pangolin detected 0.982, 0.948, 0.667, 0.487 and 0.330. Sensitivity depends strongly on the exon, precision is not measured at these thresholds, and HAL and S-Cap score only part of the variants. A curated MLH1 column is recorded but judged outside scope, because its labels include patient RNA and a pathogenicity rule. Riepe et al. (2021) tested up to 11 tools on ABCA4 and MYBPC3 variants assayed in mini- or midigenes. MCC for SpliceAI was 0.42 for ABCA4 noncanonical splice sites, 0.84 for ABCA4 deep-intronic variants and 0.35 for MYBPC3, where the Alamut 3/4 consensus reached 0.53. Those variants were chosen for testing with Alamut or MaxEntScan, and the cutoffs were tuned on the same data. Jaganathan et al. (2026, preprint), the SpliceAI2 developers at Illumina, report Spearman correlation between predicted and measured exon-inclusion change for 283,121 single-nucleotide substitutions in 599 exons of the OpenSplice reporter assay: 0.59 for SpliceAI2, 0.55 for SpliceAI, 0.53 for Pangolin and 0.49 for AlphaGenome (Extended Data Fig. 2). The comparator scores are those released with OpenSplice, while SpliceAI2 was scored by its developers, and a rank correlation over all substitutions is not precision among variants chosen for an experiment. None of these sources measures how many variants a tool would select for a new experiment and how many of those then prove splice-disruptive, so all are proxy evidence for that choice. All values are transcribed from the publications, not reproduced. Reporter and minigene endpoints do not establish patient-RNA effects or pathogenicity, and results from different sources must not be pooled. This summary describes evidence and does not recommend a method. (splicing-follow-up-20261009-source-znabu2026: Section 4 results table; splicing-follow-up-20261009-source-smith2023-table-s2: Table S2, 10% call-rate sheet; splicing-follow-up-20261009-source-smith2023: Methods; splicing-follow-up-20261009-source-riepe2021: Tables 3 to 5; rna-splicing-20261013-spliceai2-source-article: Extended Data Fig. 2 panel 'OpenSplice MPRA (ρ)', Methods 'Variant effect benchmarks')
• Relevance of BRCA1 saturation genome editing, synonymous and intronic SNVs: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) to "Prioritise variants for splicing experiments": Assessed by BRCA1 saturation genome editing, synonymous and intronic SNVs: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) (splicing-follow-up-20261009-source-smith2023-table-s2: Sheet 'Sensitivity 10% SDV', column 'BRCA1'; sheet 'Thresholds' column C; splicing-follow-up-20261009-source-smith2023: Results 'Benchmarking in the context of genome-wide prediction'; Methods)
• Relevance of FAS exon 6 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) to "Prioritise variants for splicing experiments": Assessed by FAS exon 6 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) (splicing-follow-up-20261009-source-smith2023-table-s2: Sheet 'Sensitivity 10% SDV', column 'FAS'; sheet 'Thresholds' column C; splicing-follow-up-20261009-source-smith2023: Results 'Benchmarking in the context of genome-wide prediction'; Methods)
• Relevance of MLH1 curated clinical splicing variants: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) to "Prioritise variants for splicing experiments": Assessed by MLH1 curated clinical splicing variants: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) (splicing-follow-up-20261009-source-smith2023-table-s2: Sheet 'Sensitivity 10% SDV', column 'MLH1'; sheet 'Thresholds' column C; splicing-follow-up-20261009-source-smith2023: Results 'Benchmarking in the context of genome-wide prediction'; Methods)
• Relevance of MST1R (RON) exon 11 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) to "Prioritise variants for splicing experiments": Assessed by MST1R (RON) exon 11 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) (splicing-follow-up-20261009-source-smith2023-table-s2: Sheet 'Sensitivity 10% SDV', column 'RON'; sheet 'Thresholds' column C; splicing-follow-up-20261009-source-smith2023: Results 'Benchmarking in the context of genome-wide prediction'; Methods)
• Relevance of POU1F1 exon 2 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) to "Prioritise variants for splicing experiments": Assessed by POU1F1 exon 2 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) (splicing-follow-up-20261009-source-smith2023-table-s2: Sheet 'Sensitivity 10% SDV', column 'POU1F1'; sheet 'Thresholds' column C; splicing-follow-up-20261009-source-smith2023: Results 'Benchmarking in the context of genome-wide prediction'; Methods)
• Relevance of WT1 exon 9 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) to "Prioritise variants for splicing experiments": Assessed by WT1 exon 9 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2) (splicing-follow-up-20261009-source-smith2023-table-s2: Sheet 'Sensitivity 10% SDV', column 'WT1'; sheet 'Thresholds' column C; splicing-follow-up-20261009-source-smith2023: Results 'Benchmarking in the context of genome-wide prediction'; Methods)
• reported_summary: splicing-follow-up-20261009-protocol-smith2023-brca1-sge-tn10: Reported summary Transcriptome-normalised sensitivity varied widely between algorithms, but SpliceAI, ConSpliceML, and Pangolin emerged as consistent leaders (median across datasets of 87.3%, 85.8%, and 79.9%, respectively). (Results 'Benchmarking in the context of genome-wide prediction' paragraph 2)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.