rewirebio.iobenchmarks

Use caseDNA and genomesResearch and clinical research

Prioritise variants for splicing experiments

Which evaluated configurations can inform selection of human SNVs for follow-up splicing experiments?

What the evidence shows

Three independent comparisons score several splice predictors against experimentally measured splicing of human SNVs; none of their authors developed the tools compared.

A fourth is reported by a tool's developers. Znabu et al. (2026, preprint) scored all 27,733 labelled MFASS reporter SNVs, of which 1,050 disrupt splicing. Average precision was 0.421 for Pangolin (bootstrap 95% interval 0.386 to 0.454), 0.321 for SpliceAI, 0.317 for SpliceTransformer, 0.256 for MMSplice and 0.228 for SPANR, with AUROC 0.888, 0.819, 0.786, 0.758 and 0.748.

SpliceAI was given a score of zero where it returns none. This is a different MFASS population from the matched and v2 held-out protocols and is not pooled with them. Smith and Kitzman (2023) set each tool's threshold so that it flags 10% of 500,000 random exonic and near-exonic SNVs, then measured sensitivity on saturation assays, two of which their own group produced.

SpliceAI detected 0.982 of splice-disruptive variants in BRCA1 saturation genome editing, 0.885 in POU1F1 exon 2, 0.860 in WT1 exon 9, 0.539 in FAS exon 6 and 0.237 in MST1R exon 11; Pangolin detected 0.982, 0.948, 0.667, 0.487 and 0.330. Sensitivity depends strongly on the exon, precision is not measured at these thresholds, and HAL and S-Cap score only part of the variants. A curated MLH1 column is recorded but judged outside scope, because its labels include patient RNA and a pathogenicity rule.

Riepe et al. (2021) tested up to 11 tools on ABCA4 and MYBPC3 variants assayed in mini- or midigenes. MCC for SpliceAI was 0.42 for ABCA4 noncanonical splice sites, 0.84 for ABCA4 deep-intronic variants and 0.35 for MYBPC3, where the Alamut 3/4 consensus reached 0.53. Those variants were chosen for testing with Alamut or MaxEntScan, and the cutoffs were tuned on the same data.

Jaganathan et al. (2026, preprint), the SpliceAI2 developers at Illumina, report Spearman correlation between predicted and measured exon-inclusion change for 283,121 single-nucleotide substitutions in 599 exons of the OpenSplice reporter assay: 0.59 for SpliceAI2, 0.55 for SpliceAI, 0.53 for Pangolin and 0.49 for AlphaGenome (Extended Data Fig. 2). The comparator scores are those released with OpenSplice, while SpliceAI2 was scored by its developers, and a rank correlation over all substitutions is not precision among variants chosen for an experiment. None of these sources measures how many variants a tool would select for a new experiment and how many of those then prove splice-disruptive, so all are proxy evidence for that choice.

All values are transcribed from the publications, not reproduced. Reporter and minigene endpoints do not establish patient-RNA effects or pathogenicity, and results from different sources must not be pooled. This summary describes evidence and does not recommend a method.

You bring
  • Human single nucleotide variants with alleles, genome assembly and gene/transcript context
  • The intended experimental endpoint and the number of variants that can be followed up
You want
A sourced set of exact evaluated configurations and remaining validation needs; no patient-level variant classification or recommendation.
Evidence in this release
8 comparisons shown · 28 tools · 44 configurations
Kept up to date
Evidence last checked 14 October 2026 · re-checked monthly, next due 14 November 2026

Baseline

MaxEntScan. MaxEntScan scores splice-site strength with a maximum-entropy model of the donor and acceptor motifs and is a long-standing tool in diagnostic splice prediction. Splicing assay benchmarks compare deep-learning predictors with it.

Machine-learning methods are compared with the baseline on the same data in one table: ABCA4 and MYBPC3 mini- and midigene variant sets (Riepe et al. 2021). Baseline rows are marked in each table.

Tools compared

Comparisons

Each table is one study or protocol with its own truth set and scoring. Compare tools within a table, not across tables.

MFASS v2

Proxy evidenceRewire run

Historical corrected MFASS v2 ranking and top-100 follow-up precision for supervised feature and training-prior controls

Sorted by AUROC (higher is better). The best value in each column is highlighted.

Scroll sideways to see all 3 columns.

ToolAUROCAverage precisionPrecision at 100
Corrected k-mer / position baseline0.778 (best in column)0.2864 (best in column)0.61 (best in column)
Training class prior / majority (MFASS v2 canonical held-out split)0.5000.03780.04
Caveats and method
  • Historical v2 annotation/coverage differs from matched-annotation v1: do not combine measurements.
  • Each method uses its own scored subset; marginal point scores do not establish a matched winner.
  • The constant prior has tied predictions; top-100 positives depend on fixed label-independent tie order.
  • Assay labels are not patient RNA or clinical endpoints.
  • DNABERT2 is represented as a pipeline, which is ineligible for this exact-configuration use-case mapping. Its existing evaluation remains in the catalogue and coverage audit; no record was promoted or score duplicated.

Protocol: MFASS v2

Feng Acceptor sequence classification

Proxy evidenceIndependent study

Acceptor sequence-classification AUC for five DNA foundation-model embeddings

Sorted by AUROC (higher is better). The best value in each column is highlighted.

ToolAUROC (task: splice acceptor)
DNABERT-2foundation model0.8969 (best in column)
Caduceus-Phfoundation model0.8449
GROVERfoundation model0.8041
HyenaDNAfoundation model0.7946
NT-v2foundation model0.7928
Caveats and method
  • Sequence-label acceptor classification is not a minigene reporter exon-inclusion endpoint and is not patient-RNA splicing validation.
  • Exact per-task split sizes/counts are unextracted from the inspected main text (Supplementary Data 6 is identified but not inspected).
  • No independent reproduction or qualified human scientific review.

Protocol: Feng Acceptor sequence classification

Feng Donor sequence classification

Proxy evidenceIndependent study

Donor sequence-classification AUC for five DNA foundation-model embeddings

Sorted by AUROC (higher is better). The best value in each column is highlighted.

ToolAUROC (task: splice donor)
DNABERT-2foundation model0.9056 (best in column)
Caduceus-Phfoundation model0.8535
NT-v2foundation model0.8198
GROVERfoundation model0.8190
HyenaDNAfoundation model0.8128
Caveats and method
  • Sequence-label donor classification is not a minigene reporter exon-inclusion endpoint and is not patient-RNA splicing validation.
  • Exact per-task split sizes/counts are unextracted from the inspected main text (Supplementary Data 6 is identified but not inspected).
  • No independent reproduction or qualified human scientific review.

Protocol: Feng Donor sequence classification

OpenSplice MPRA delta PSI (SpliceAI2 paper, developer-reported)

Proxy evidenceAuthor-reported

Spearman correlation between predicted and measured exon-inclusion change (delta PSI) for 283,121 single-nucleotide substitutions across 599 exons in the OpenSplice MPRA, for SpliceAI, Pangolin, AlphaGenome and SpliceAI2

Spearman (higher is better), with the other metrics below each value. The best value in each column is highlighted.

ToolAll substitutions and exon or intron strata
SpliceAI2machine learning0.59 (best in column)allSpearman (exonic) 0.62Spearman (intronic) 0.52
SpliceAImachine learning0.55allSpearman (exonic) 0.57Spearman (intronic) 0.50
Pangolinmachine learning0.53allSpearman (exonic) 0.54Spearman (intronic) 0.49
AlphaGenomemachine learning0.49allSpearman (exonic) 0.52Spearman (intronic) 0.44
Caveats and method
  • Developer-reported; the OpenSplice dataset is from another group (reference 23), but this comparison was run by the SpliceAI2 authors.
  • Spearman over all substitutions; no precision at a follow-up capacity and no uncertainty printed.
  • Training overlap: SpliceAI2 was trained on all human genes outside chromosomes 21 and 22, including the reference splicing of most OpenSplice exons' genes.
  • The use case's baseline, MaxEntScan, is not evaluated.
  • Comparator scores come from the OpenSplice release; this paper does not describe their settings.
  • Preprint (bioRxiv v1), not peer reviewed.

Protocol: OpenSplice MPRA delta PSI prediction for 283,121 single-nucleotide substitutions in 599 exons (SpliceAI2 paper, Spearman)

ABCA4 and MYBPC3 mini- and midigene variant sets (Riepe et al. 2021)

Proxy evidenceIndependent study

MCC (higher is better), with the other metrics below each value. The best value in each column is highlighted.

Scroll sideways to see all 3 columns.

ToolABCA4 noncanonical splice-site variantsABCA4 deep-intronic variantsMYBPC3 noncanonical splice-site variants
SpliceAI (Riepe et al. 2021)machine learning0.42 (best in column)Accuracy 79%False negatives 14False positives 1Count (variants without a score) 0
6 more valuesNegative predictive value 30%P 98%R 78%Sp 86%True negatives 6True positives 50
0.84 (best in column)Accuracy 94%False negatives 2False positives 3Count (variants without a score) 0
6 more valuesNegative predictive value 97%P 86%R 90%Sp 95%True negatives 57True positives 19
0.35Accuracy 67%False negatives 12False positives 8Count (variants without a score) 0
6 more valuesNegative predictive value 61%P 73%R 65%Sp 70%True negatives 19True positives 22
SpliceRover (Riepe et al. 2021)machine learning0.39Accuracy 76%False negatives 16False positives 1Count (variants without a score) 0
6 more valuesNegative predictive value 27%P 98%R 75%Sp 86%True negatives 6True positives 48
0.44Accuracy 75%False negatives 6False positives 14Count (variants without a score) 0
6 more valuesNegative predictive value 88%P 52%R 71%Sp 77%True negatives 46True positives 15
0.31Accuracy 66%False negatives 12False positives 9Count (variants without a score) 0
6 more valuesNegative predictive value 60%P 71%R 65%Sp 67%True negatives 18True positives 22
DSSP (Riepe et al. 2021)machine learning0.35Accuracy 79%False negatives 13False positives 2Count (variants without a score) 0
6 more valuesNegative predictive value 28%P 96%R 80%Sp 71%True negatives 5True positives 51
0.27Accuracy 67%False negatives 8False positives 19Count (variants without a score) 0
6 more valuesNegative predictive value 84%P 41%R 62%Sp 68%True negatives 41True positives 13
0.28Accuracy 64%False negatives 12False positives 10Count (variants without a score) 0
6 more valuesNegative predictive value 59%P 69%R 65%Sp 63%True negatives 17True positives 22
MMSplice (Riepe et al. 2021)machine learning0.24Accuracy 68%False negatives 21False positives 2Count (variants without a score) 0
6 more valuesNegative predictive value 19%P 96%R 67%Sp 71%True negatives 5True positives 43
Not reported0.51Accuracy 75%False negatives 9False positives 6Count (variants without a score) 0
6 more valuesNegative predictive value 70%P 81%R 74%Sp 78%True negatives 21True positives 25
Spidex (Riepe et al. 2021)machine learning0.24Accuracy 68%False negatives 21False positives 2Count (variants without a score) 5
6 more valuesNegative predictive value 19%P 96%R 67%Sp 71%True negatives 5True positives 43
Not reported0.25Accuracy 62%False negatives 14False positives 9Count (variants without a score) 3
6 more valuesNegative predictive value 56%P 69%R 59%Sp 67%True negatives 18True positives 20
NNSPLICE (Riepe et al. 2021)machine learning0.23Accuracy 66%False negatives 22False positives 2Count (variants without a score) 0
6 more valuesNegative predictive value 19%P 95%R 66%Sp 71%True negatives 5True positives 42
0.35Accuracy 70%False negatives 7False positives 17Count (variants without a score) 0
6 more valuesNegative predictive value 86%P 45%R 67%Sp 72%True negatives 43True positives 14
0.45Accuracy 72%False negatives 11False positives 6Count (variants without a score) 0
6 more valuesNegative predictive value 66%P 79%R 68%Sp 78%True negatives 21True positives 23
MaxEntScan (Riepe et al. 2021)Baselineconventional0.21Accuracy 63%False negatives 24False positives 2Count (variants without a score) 0
6 more valuesNegative predictive value 17%P 95%R 63%Sp 71%True negatives 5True positives 40
0.24Accuracy 64%False negatives 8False positives 21Count (variants without a score) 0
6 more valuesNegative predictive value 83%P 38%R 62%Sp 65%True negatives 39True positives 13
0.48Accuracy 74%False negatives 10False positives 6Count (variants without a score) 0
6 more valuesNegative predictive value 68%P 80%R 71%Sp 78%True negatives 21True positives 24
SpliceSiteFinder-like (Riepe et al. 2021)conventional0.21Accuracy 63%False negatives 24False positives 2Count (variants without a score) 0
6 more valuesNegative predictive value 17%P 95%R 63%Sp 71%True negatives 5True positives 40
0.06Accuracy 54%False negatives 10False positives 27Count (variants without a score) 0
6 more valuesNegative predictive value 77%P 29%R 52%Sp 55%True negatives 33True positives 11
0.51Accuracy 75%False negatives 9False positives 6Count (variants without a score) 0
6 more valuesNegative predictive value 70%P 81%R 74%Sp 78%True negatives 21True positives 25
Alamut Consensus 3/4 (Riepe et al. 2021)conventional0.16Accuracy 56%False negatives 29False positives 2Count (variants without a score) 0
6 more valuesNegative predictive value 15%P 95%R 55%Sp 71%True negatives 5True positives 35
0.28Accuracy 70%False negatives 10False positives 14Count (variants without a score) 0
6 more valuesNegative predictive value 82%P 44%R 52%Sp 77%True negatives 46True positives 11
0.53 (best in column)Accuracy 75%False negatives 11False positives 4Count (variants without a score) 0
6 more valuesNegative predictive value 68%P 85%R 68%Sp 85%True negatives 23True positives 23
CADD (Riepe et al. 2021)machine learning0.12Accuracy 62%False negatives 24False positives 3Count (variants without a score) 0
6 more valuesNegative predictive value 14%P 93%R 63%Sp 57%True negatives 4True positives 40
0.15Accuracy 59%False negatives 9False positives 24Count (variants without a score) 0
6 more valuesNegative predictive value 80%P 33%R 57%Sp 60%True negatives 36True positives 12
0.32Accuracy 66%False negatives 13False positives 8Count (variants without a score) 0
6 more valuesNegative predictive value 59%P 72%R 62%Sp 70%True negatives 19True positives 21
GeneSplicer (Riepe et al. 2021)machine learning0.03Accuracy 49%False negatives 33False positives 3Count (variants without a score) 0
6 more valuesNegative predictive value 11%P 91%R 48%Sp 57%True negatives 4True positives 31
0.36Accuracy 72%False negatives 7False positives 16Count (variants without a score) 0
6 more valuesNegative predictive value 86%P 47%R 67%Sp 73%True negatives 44True positives 14
0.51Accuracy 75%False negatives 9False positives 6Count (variants without a score) 0
6 more valuesNegative predictive value 70%P 81%R 74%Sp 78%True negatives 21True positives 25

R recall, P precision, Sp specificity.

Caveats and method

What each column measures:

  • ABCA4 noncanonical splice-site variants: Confusion matrix, accuracy, PPV, sensitivity, specificity, NPV and MCC for up to 11 splice predictors on 71 ABCA4 noncanonical splice-site variants tested in mini- or midigene assays
  • ABCA4 deep-intronic variants: Confusion matrix, accuracy, PPV, sensitivity, specificity, NPV and MCC for up to 11 splice predictors on 81 ABCA4 deep-intronic variants tested in mini- or midigene assays
  • MYBPC3 noncanonical splice-site variants: Confusion matrix, accuracy, PPV, sensitivity, specificity, NPV and MCC for up to 11 splice predictors on 61 MYBPC3 noncanonical splice-site variants tested in mini- or midigene assays
  • Variants were selected for testing with Alamut tools (ABCA4) or MaxEntScan (MYBPC3), which can favour those tools and their relatives.
  • Cutoffs appear to be tuned on the same data set (Table 2 ROC-optimal thresholds); Tables 3-5 do not state the cutoff, so values may be optimistic.
  • Small single-gene sets with strong class imbalance (ABCA4 NCSS 90% positive, ABCA4 DI 26% positive).
  • Variants are from patients but splicing was measured in mini- or midigene assays in HEK293T, not in patient tissue.
  • The authors note the selection bias but observed the opposite pattern (Alamut 3/4 best on MYBPC3).
  • Spidex's unscored variants are counted in its confusion matrix without a stated class.

Protocols: ABCA4 noncanonical splice-site variants: classification against mini- or midigene splicing results (Riepe et al. Table 3), ABCA4 deep-intronic variants: classification against mini- or midigene splicing results (Riepe et al. Table 4), MYBPC3 noncanonical splice-site variants: classification against mini- or midigene splicing results (Riepe et al. Table 5)

Saturation splicing assays, sensitivity at a 10% genome-wide call rate (Smith and Kitzman 2023)

Proxy evidenceIndependent study

Recall (higher is better), with the other metrics below each value. The best value in each column is highlighted.

Scroll sideways to see all 5 columns.

ToolBRCA1 saturation genome editing, synonymous and intronic SNVsFAS exon 6 saturation MPSAPOU1F1 exon 2 saturation MPSAMST1R (RON) exon 11 saturation MPSAWT1 exon 9 saturation MPSA
Pangolin (Smith and Kitzman 2023)machine learning98.2% (best in column)all variantsR (exonic variants) 90.9%R (intronic variants) 98.6%R (intronic variants excluding essential splice sites) 96.9%48.70%allR 48.7%94.8% (best in column)all variantsR (exonic variants) 94.3%R (intronic variants) 96.2%R (intronic variants excluding essential splice sites) 95%33.01% (best in column)all variantsR (exonic variants) 35.2%R (intronic variants) 28.9%R (intronic variants excluding essential splice sites) 23.5%66.7%all variantsR (exonic variants) 36%R (intronic variants) 90.6%R (intronic variants excluding essential splice sites) 85%
SpliceAI (Smith and Kitzman 2023)machine learning98.2% (best in column)all variantsR (exonic variants) 90.9%R (intronic variants) 98.6%R (intronic variants excluding essential splice sites) 96.9%53.91%allR 53.9%88.5%all variantsR (exonic variants) 90%R (intronic variants) 84.6%R (intronic variants excluding essential splice sites) 80%23.72%all variantsR (exonic variants) 24.0%R (intronic variants) 23.2%R (intronic variants excluding essential splice sites) 17.4%86.0%all variantsR (exonic variants) 80%R (intronic variants) 90.6%R (intronic variants excluding essential splice sites) 85%
MMSplice (Smith and Kitzman 2023)machine learning97.8%all variantsR (exonic variants) 81.8%R (intronic variants) 98.6%R (intronic variants excluding essential splice sites) 96.9%36.52%allR 36.5%41.1%all variantsR (exonic variants) 35.7%R (intronic variants) 56%R (intronic variants excluding essential splice sites) 42.1%4.65%all variantsR (exonic variants) 3.37%R (intronic variants) 7.04%R (intronic variants excluding essential splice sites) 7.58%71.9%all variantsR (exonic variants) 40%R (intronic variants) 96.9%R (intronic variants excluding essential splice sites) 95%
S-Cap (Smith and Kitzman 2023)machine learning95.5%all variantsR (exonic variants) 100%R (intronic variants) 95.3%R (intronic variants excluding essential splice sites) 89.6%0.00%allR 0%21.6%all variantsR (exonic variants) 18.2%R (intronic variants) 23.1%R (intronic variants excluding essential splice sites) 0%13.33%all variantsR (exonic variants) 11.6%R (intronic variants) 14.2%R (intronic variants excluding essential splice sites) 7.63%95.2%all variantsR (exonic variants) 90%R (intronic variants) 96.9%R (intronic variants excluding essential splice sites) 95%
ConSpliceML (Smith and Kitzman 2023)machine learning91.5%all variantsR (exonic variants) 72.7%R (intronic variants) 92.5%R (intronic variants excluding essential splice sites) 83.3%5.22%allR 5.22%80.2%all variantsR (exonic variants) 85.7%R (intronic variants) 65.4%R (intronic variants excluding essential splice sites) 55%26.16%all variantsR (exonic variants) 29.2%R (intronic variants) 20.4%R (intronic variants excluding essential splice sites) 14.4%96.5% (best in column)all variantsR (exonic variants) 100%R (intronic variants) 93.75%R (intronic variants excluding essential splice sites) 90%
HAL (Smith and Kitzman 2023)machine learning90.9%allR 90.9%14.78%allR 14.8%75.0%allR 75%16.48%allR 16.5%24.0%allR 24%
SPANR (Smith and Kitzman 2023)machine learning75.3%all variantsR (exonic variants) 27.3%R (intronic variants) 77.8%R (intronic variants excluding essential splice sites) 58.3%57.39% (best in column)allR 57.4%58.3%all variantsR (exonic variants) 51.4%R (intronic variants) 76.9%R (intronic variants excluding essential splice sites) 70%10.27%all variantsR (exonic variants) 10.5%R (intronic variants) 9.86%R (intronic variants excluding essential splice sites) 3.03%91.2%all variantsR (exonic variants) 96%R (intronic variants) 87.5%R (intronic variants excluding essential splice sites) 80%
SQUIRLS (Smith and Kitzman 2023)machine learning65.5%all variantsR (exonic variants) 45.5%R (intronic variants) 66.5%R (intronic variants excluding essential splice sites) 59.4%5.26%allR 5.26%91.6%all variantsR (exonic variants) 90%R (intronic variants) 96%R (intronic variants excluding essential splice sites) 95%11.49%all variantsR (exonic variants) 8.24%R (intronic variants) 17.6%R (intronic variants excluding essential splice sites) 13.6%96.2%all variantsR (exonic variants) 100%R (intronic variants) 92.9%R (intronic variants excluding essential splice sites) 88.9%

R recall.

Caveats and method

What each column measures:

  • BRCA1 saturation genome editing, synonymous and intronic SNVs: Sensitivity for BRCA1 saturation genome editing, synonymous and intronic SNVs splice-disruptive SNVs at thresholds calling 10% of 500,000 background SNVs, for HAL, S-Cap, MMSplice, SQUIRLS, SPANR, SpliceAI, Pangolin and ConSpliceML, overall and by variant class
  • FAS exon 6 saturation MPSA: Sensitivity for FAS exon 6 saturation MPSA splice-disruptive SNVs at thresholds calling 10% of 500,000 background SNVs, for HAL, S-Cap, MMSplice, SQUIRLS, SPANR, SpliceAI, Pangolin and ConSpliceML, overall and by variant class
  • POU1F1 exon 2 saturation MPSA: Sensitivity for POU1F1 exon 2 saturation MPSA splice-disruptive SNVs at thresholds calling 10% of 500,000 background SNVs, for HAL, S-Cap, MMSplice, SQUIRLS, SPANR, SpliceAI, Pangolin and ConSpliceML, overall and by variant class
  • MST1R (RON) exon 11 saturation MPSA: Sensitivity for MST1R (RON) exon 11 saturation MPSA splice-disruptive SNVs at thresholds calling 10% of 500,000 background SNVs, for HAL, S-Cap, MMSplice, SQUIRLS, SPANR, SpliceAI, Pangolin and ConSpliceML, overall and by variant class
  • WT1 exon 9 saturation MPSA: Sensitivity for WT1 exon 9 saturation MPSA splice-disruptive SNVs at thresholds calling 10% of 500,000 background SNVs, for HAL, S-Cap, MMSplice, SQUIRLS, SPANR, SpliceAI, Pangolin and ConSpliceML, overall and by variant class
  • Sensitivity only: the threshold fixes a genome-wide call rate, not specificity on the benchmark, so precision is not measured.
  • Per-dataset SDV and neutral counts are not printed in Table S2.
  • Blank cells (HAL intronic rows, FAS intronic columns) are not results and are not stored.
  • The masking setting behind Table S2 is not stated for SpliceAI or Pangolin; both masked and unmasked scores were computed.
  • HAL and S-Cap score only some variants, so their thresholds rest on part of the background; HAL's all-variant values equal its exonic values.
  • SpliceAI's threshold comes from version 1.3 precomputed background scores with unstated distance and masking; benchmark variants were scored with 1.3.1.

Protocols: BRCA1 saturation genome editing, synonymous and intronic SNVs: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2), FAS exon 6 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2), POU1F1 exon 2 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2), MST1R (RON) exon 11 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2), WT1 exon 9 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)

MFASS full labelled cohort (Znabu et al. 2026 preprint)

Proxy evidenceIndependent study

AUROC and average precision (with bootstrap 95% intervals) for Pangolin, SpliceAI, SpliceTransformer, MMSplice and SPANR on all 27,733 labelled MFASS SNVs

Sorted by Average precision (higher is better). The best value in each column is highlighted.

Scroll sideways to see all 3 columns.

ToolAverage precisionAUROCCount (variants scored)
Pangolin (Znabu et al. 2026)machine learning0.421 (best in column)0.888 (best in column)27,733
SpliceAI (Znabu et al. 2026)machine learning0.3210.81927,733
SpliceTransformer (Znabu et al. 2026)machine learning0.3170.78627,733
MMSplice (Znabu et al. 2026)machine learning0.2560.75827,733
SPANR (legacy) (Znabu et al. 2026)machine learning0.2280.74827,663
Caveats and method
  • Preprint, not peer reviewed; independent authors.
  • Whole labelled cohort, not a held-out split; SPANR scored 27,663 variants.
  • The 27,733 count matches rewire-mfass-v2-dataset; variant-level identity was not checked.
  • Reporter exon inclusion, not patient RNA or pathogenicity.
  • SpliceAI was given a score of 0 where it returns none (outside annotated genes); the authors report the same ranking on the 12,855 variants SpliceAI scores above zero.
  • MMSplice was scored on a reconstructed cassette context, which the authors say likely understates it.

Protocol: MFASS splice-disrupting variant detection, all 27,733 labelled SNVs (Znabu et al. 2026)

MFASS: matched GENCODE 44 canonical annotation

Proxy evidenceRewire run

Ranking held-out MFASS SNVs by reporter-assay splice disruption under matched canonical annotation.

Sorted by AUROC (higher is better). The best value in each column is highlighted.

Scroll sideways to see all 4 columns.

ToolAUROCAverage precisionPrecision at 100Recall at 100
Pangolin 1.0.2 + per-gene masking patch · mask False (P0)machine learning0.876 (best in column)0.3890.650.207
Pangolin 1.0.2 + per-gene masking patch · mask True (P1)machine learning0.8730.411 (best in column)0.66 (best in column)0.210 (best in column)
SpliceAI 1.3.1 · mask 1 (S1)machine learning0.8150.3130.650.207
SpliceAI 1.3.1 · mask 0 (S0)machine learning0.8040.2950.630.201
Caveats and method
  • 23 assembly-orientation mismatches and four canonical-transcript-span exclusions remain unscored. The latter are protocol exclusions, not established faulty variants; full-population performance is unknown.
  • No top-100 precision difference is established and masked Pangolin is tie-sensitive. Paired contrast intervals are not individual-condition intervals.
  • Exploratory source review only; no independent human review, replication or clinical validation. Assembly-issue author confirmation is not established.

Protocol: MFASS: matched GENCODE 44 canonical annotation

Not covered yet

  • All four conditions scored 8,297 of 8,324 held-out variants. The same 27 exclusions comprise 23 hg19-to-hg38 assembly-orientation mismatches and four canonical-transcript-span exclusions; the latter are not established faulty variants. Missing predictions are not negative predictions.
  • No top-100 precision difference is established; Pangolin's masked top-100 result is sensitive to the registered tie order. Precision at 100 does not transfer automatically to another follow-up capacity or prevalence.
  • Individual-condition uncertainty intervals are not recorded. Paired-contrast intervals concern differences between conditions and must not be shown as each condition's uncertainty.
  • The study is exploratory: prior outcomes were inspected and nine contrast intervals are unadjusted. Matching annotation does not isolate model architecture.
  • Author confirmation of the assembly-orientation finding is not established. Human scientific review and independent replication remain outstanding.
  • These mappings do not change the MFASS task's discovered status. A source-reviewed applicability mapping is separate from reviewing the task record.
  • 30 September 2026 audit: The four matched conditions exclude the same 27 of 8,324 held-out variants; missing scores are not negative predictions.
  • 30 September 2026 audit: Historical v2 marginal scores have different scored subsets/annotations and require their own paired comparisons. Constant-prior top-100 counts reflect tied-score order. DNABERT2 is a pipeline rather than an exact configuration and remains outside the active mapping.
  • 30 September 2026 audit: MFASS reporter exon inclusion does not validate patient RNA, diagnosis or prospective clinical follow-up. Human scientific review and external reproduction remain outstanding.
  • No linked source measures precision among the variants a tool would select for a new splicing experiment. The sources report sensitivity at a genome-wide background call rate (Smith and Kitzman 2023), classification at cutoffs tuned on the same data (Riepe et al. 2021), or ranking metrics on a whole assay cohort (Znabu et al. 2026).
  • Smith and Kitzman 2023 report sensitivity at transcriptome-normalised thresholds; precision and prAUC per tool and dataset are figure-only (Figure 3), and the 5% and 20% call-rate sheets and the sensitivity-AUC sheet of Table S2 are not extracted.
  • Riepe et al. 2021 Tables 3 to 5 do not state the cutoff applied; Table 2 cutoffs were optimised on the same data sets, so values may be optimistic.
  • Znabu et al. 2026 is a preprint scoring the whole labelled MFASS cohort; it is not comparable with the matched-annotation or v2 held-out MFASS protocols.
  • CAGI5 Vex-seq and MaPSy assessments (Mount et al. 2019) print per-submission values for challenge entries, most of which are not released tools; not extracted.
  • No independent printed comparison on Vex-seq or MaPSy for current tools (SpliceAI, Pangolin, SpliceTransformer) was found.
  • OpenSplice (bioRxiv 2026.05.22.727141) and SeqSplice BRCA1/BRCA2 (Genome Res 2025) are leads not screened for per-tool tables.

Contribute evidence or propose a correction

Question, sources and review

Who this is for, setting and exclusions

Inspect the configurations evaluated against experimentally measured splicing of human SNVs (MFASS reporter assays, saturation minigene and genome-editing assays, and mini- or midigene tests), with their limitations, before selecting a method and designing validation in the intended experimental setting.

Who this is for
  • Experimental researchers selecting variants for functional follow-up
  • Computational researchers comparing splice-effect configurations
  • Clinical researchers investigating the limits of assay evidence
Biological setting
Each linked comparison scores splice predictors against splicing measured in an experiment. MFASS measures exon recognition of human SNVs in an artificial minigene reporter; the matched study evaluates genomic-context SpliceAI and Pangolin configurations against this endpoint on a fixed held-out population. Saturation minigene assays and BRCA1 saturation genome editing measure every SNV in and around single exons, and mini- or midigene tests in HEK293T cells assess selected ABCA4 and MYBPC3 variants. Each source reports its own population, thresholds and metrics, and results are not pooled across sources.
Clinical research scope
Clinical applicability is not established. Reporter-assay ranking does not demonstrate patient-RNA performance, pathogenicity classification or clinical yield; validation in the intended population and workflow is still needed.

Outside this use case

  • Indels and variant types outside the reviewed human SNV protocol
  • Patient-RNA effects, disease pathogenicity, diagnostic yield and treatment decisions
  • Pooling the matched annotation study with historical MFASS runs using other annotations or scoring populations
Related articles

Read the background, methods and practical considerations behind this question.

Sources, review and downloads

Automated source review · 2026-09-30 · Codex source curation, independent worker cross-review and root integration review

Source-backed literature curation with independent automated transcription and scope checks. No new model execution, independent experimental replication, qualified human scientific review or clinical validation.

Release 2026-11-02-a7f03c9e55a5 · question use-case-splicing-follow-up

Use-case input digest 43a80aef5473f1a49c013b6d46b720b850ff5a0d18dcbb741dcdeed1ed082db1

Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)