Analyzing the performance of deep learning splice prediction algorithms (Fortier, Rudy and Scherer, PLOS One 2026)

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 2 models
• 1 method
• 4 configurations
• 6 protocols
• 6 datasets
• 24 evaluations
• 96 results
• 7 evidence claims

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• Baseline: Prioritise variants for splicing experiments: Baseline MaxEntScan (splicing-follow-up-20261009-source-riepe2021: Introduction (MaxEntScan apart from the classical machine learning tools), Table 1 and Methods (MaxEntScan selection criterion); rna-splicing-20261011-fortier2026-source-article: Introduction (MaxEntScan commonly used))
• Baseline: Select a patient-RNA splicing-variant validation workflow: Baseline FRASER and OUTRIDER, with a legacy splice-site vote (amp-oncology-rna-20261007-source-pmc7822922: Results (beta-binomial outlier test, denoising autoencoder, DROP workflow); rna-splicing-20261009-source-segers2026: Background (FRASER as state of the art, and OUTRIDER, OutSingle and FRASER); rna-splicing-20261011-fortier2026-source-article: Methods (Legacy Ensemble N-of-4 vote) and Introduction (NNSplice and machine learning))
• Evidence summary: patient RNA splicing validation: Summary The one recovery result already on this page is FRASER on the 119-sample Kremer fibroblast cohort, reported by its developers: a mean 85% (11 of 13) of known pathogenic splicing events recovered at a 30-sample subsample. Six published comparisons add evidence. Segarra-Casas et al. (2025), independent of the tool developers, recorded which callers reported 16 known pathogenic splicing events in clinical muscle RNA-seq from 98 samples: FRASER with |dPSI| > 0.1 reported 13, FRASER with |dPSI| > 0.3 reported 9, FRASER2 10 (and one more only with the full cohort), rMATS-turbo 7 and LeafCutterMD 4 (Table 2); the events were established partly with these callers. Segers et al. (2026), the saseR developers, ranked 12 reported disease genes in the Kremer cohort: the gene ranked within the top 10 in 9 of 12 samples for saseR-junctions, in 6 of 12 for FRASER 2.0 with the autoencoder and in 7 of 12 for FRASER 2.0 with PCA (Table 2); the 12 genes were reported in earlier analyses of the same cohort, partly with OUTRIDER and FRASER, and several are expression rather than splicing defects. Silverstein et al. (2026, preprint), a clinical research group that did not develop the splicing callers, scored them on 97 previously diagnosed neuromuscular RNA-seq samples and also counted outlier calls in true-negative samples (Table S9). On known pathogenic splicing events in muscle and fibroblast RNA-seq, rMATS-turbo detected 15 of 23 and MINTIE 8 of 23; on muscle, fibroblast and cycloheximide-treated fibroblast RNA-seq, FRASER detected 21 of 68 and LeafcutterMD none of 67; FRASER2.0, run on muscle only, detected 2 of 16. rMATS-turbo and MINTIE reported outliers in all 9 true-negative samples they were run on, FRASER and LeafcutterMD in all 21, and FRASER2.0 in 2 of 6. Denominators differ by tool, a negative sample with any outlier counts as one false positive, and the known events were chosen because they were clinically interpretable and visible in the RNA-seq, which the authors say favours clearly defined events. Drost et al. (2025), a diagnostic laboratory independent of the tool developers, scored 243 clinically ascertained variants, each tested for a splicing effect in patient cells or by exon trapping, with four DNA-based predictors. On all 243, AUROC was 0.895 for SpliceAI, 0.885 for Pangolin, 0.830 for SPiP and 0.806 for SQUIRLS (Data S1 Table S3); at literature thresholds sensitivity was 0.871, 0.858, 0.832 and 0.800 and specificity 0.750, 0.773, 0.761 and 0.591 (Table S4), and the authors' rule of requiring at least two of the four tools gave sensitivity 0.916 and specificity 0.727. Most in-house variants were sent for testing because another predictor (Alamut) flagged them, so negatives are mostly predicted-but-unconfirmed variants. Two further DNA-based comparisons add proxy evidence. Fortier et al. (2026), independent of the SpliceAI, CI-SpliceAI and OpenSpliceAI developers, report balanced accuracy on functional splice-assay variants in ABCA4 and MYBPC3 of 0.836 for SpliceAI, 0.794 for CI-SpliceAI and 0.746 for OpenSpliceAI, and on 242 deep intronic pathogenic variants with 242 common controls of 0.940, 0.890 and 0.841 (Tables 2 and 4); both sets of thresholds were tuned on the same data, so these are upper-bound estimates, and CI-SpliceAI was trained on all chromosomes. Alavi (2026, preprint) applied thresholds tuned on OPA1 variants to 76 functionally validated canonical splice-site variants and 107 ClinVar benign intronic variants from 65 genes: sensitivity and specificity were 88.2% and 99.1% for SpliceAI, 85.5% and 86.9% for pretrained NTv3 650M, 71.1% and 100% for Evo2 40B with a human identifier, 56.6% and 80.4% for AlphaGenome (either module) and 27.6% and 85.0% for Genos 10B (Table IV). Jaganathan et al. (2026, preprint), the SpliceAI2 developers at Illumina, scored four DNA-based predictors on rare variants that create a GT or AG dinucleotide in 5,435 Genomics England rare-disease participants with blood RNA-seq, against new junctions seen in the carriers' RNA (3,065 cryptic gains and 1,775,800 nulls): auPRC was 0.59 for SpliceAI2, 0.48 for SpliceAI, 0.40 for Pangolin and 0.36 for AlphaGenome, or 0.45 with AlphaGenome's raw splice site score (Fig. 4g; Methods). Only splicing visible in blood can be confirmed, so some nulls may alter splicing in other tissues, and SpliceAI2 was trained on all human genes except those on chromosomes 21 and 22. Of 23 rare variants in phenotype-matched panel genes that SpliceAI2 prioritised, 17 showed the predicted splicing change in RNA-seq; no other tool was put through this step. The paper compares no rule-based splice-site scorer. All values are transcribed from the publications, not reproduced; none is a diagnostic-yield estimate, and results from different sources measure different endpoints and must not be pooled. This summary describes evidence and does not recommend a clinical workflow. (amp-oncology-rna-20261007-source-pmc7822922: FRASER results; rna-splicing-20261009-source-segarracasas2025: Table 2, Results paragraph 3; rna-splicing-20261009-source-segers2026: Table 2; rna-splicing-20261009-source-drost2025-data-s1: Data S1 Tables S3 and S4; rna-splicing-20261009-source-drost2025: Methods; rna-splicing-20261011-silverstein2026-source-tables-s1-s10: Table S9 rows 2 to 6; rna-splicing-20261011-fortier2026-source-article: Tables 2 and 4; rna-splicing-20261011-alavi2026-source-article: Table IV; rna-splicing-20261013-spliceai2-source-article: Fig. 4g, Methods 'Evaluation of competing methods', Main 'Contribution to rare genetic disease' paragraph 4)
• Relevance of Splice-altering variant classification, Barbosa et al. deep intronic pathogenic variants with matched common controls (484 variants) (Fortier et al. Table 4) to "Select a patient-RNA splicing-variant validation workflow": Assessed by Splice-altering variant classification, Barbosa et al. deep intronic pathogenic variants with matched common controls (484 variants) (Fortier et al. Table 4) (rna-splicing-20261011-fortier2026-source-article: Methods 'Barbosa deep intronic benchmark dataset'; Results 'Barbosa deep intronic benchmark results'; Table 4)
• Relevance of Splice-altering variant classification, CI-SpliceAI benchmark variant set (1,316 variants) (Fortier et al. Table 1) to "Select a patient-RNA splicing-variant validation workflow": Assessed by Splice-altering variant classification, CI-SpliceAI benchmark variant set (1,316 variants) (Fortier et al. Table 1) (rna-splicing-20261011-fortier2026-source-article: Methods 'CI-SpliceAI benchmark dataset'; Table 1)
• Relevance of Splice-altering variant classification, Riepe et al. functional splice assay variants (213 variants, ABCA4 and MYBPC3) (Fortier et al. Table 2) to "Select a patient-RNA splicing-variant validation workflow": Assessed by Splice-altering variant classification, Riepe et al. functional splice assay variants (213 variants, ABCA4 and MYBPC3) (Fortier et al. Table 2) (rna-splicing-20261011-fortier2026-source-article: Methods 'Riepe benchmark dataset' and 'Splice prediction tools'; Table 2)
• Relevance of Splice-altering variant classification, SPiP benchmark variant set (99,601 variants) (Fortier et al. Table 3) to "Select a patient-RNA splicing-variant validation workflow": Assessed by Splice-altering variant classification, SPiP benchmark variant set (99,601 variants) (Fortier et al. Table 3) (rna-splicing-20261011-fortier2026-source-article: Methods 'SPiP benchmark dataset'; Results 'SPiP benchmark results'; Table 3)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.