rewirebio.iobenchmarks
Contested result

Pangolin and SpliceAI: Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) reverses the order

  • Low severity
  • Not yet reviewed
  • Ordering reversal
  • unknown

Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) ranks SpliceAI above Pangolin in 27 of 28 comparisons; 3 of 3 other studies rank Pangolin above SpliceAI.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

7 papers rank Pangolin above SpliceAI in 29 of their 72 comparisons and SpliceAI higher in 18; 25 are ties. Genome modelling and design across all… ranks SpliceAI above Pangolin in 27 of its 28 comparisons and Pangolin higher in 1.

Papers grouped by which system they rank higher: Pangolin or SpliceAIRank Pangolin higher: Benchmarking splice variant prediction…, Amino-acid composition + fixed ridge…, MFASS v2 pinned rewire artifacts, A Reproducible MFASS Benchmark…. Rank SpliceAI higher: Genome modelling and design across all… (disagrees). Mixed: Routine RNA-based analysis of potential…, Jaganathan et al. 2026, Cheng et al. 2026.Rank Pangolin higherMixedRank SpliceAI higherBenchmarking splicevariant prediction…Benchmarking splice variant prediction algorithms using massively parallel splicing assaysAmino-acid composition+ fixed ridge…Amino-acid composition + fixed ridge: automated execution auditMFASS v2 pinnedrewire artifactsMFASS v2 pinned rewire artifactsA ReproducibleMFASS Benchmark…A Reproducible MFASS Benchmark of Splice-Disruption Predictors Reveals a Shared Exon-Interior Blind SpotRoutine RNA-basedanalysis of potential…Routine RNA-based analysis of potential splicing variants facilitates genomic diagnostics and reveals limitations of in silico prediction toolsJaganathan et al.2026A unified framework for quantitative splicing and transcript prediction (Jaganathan et al., bioRxiv 2026)Cheng et al.2026AlphaGenome Atlas: in silico mutagenesis of the entire human genome (Cheng, Taylor, Nicolaisen et al., medRxiv 2026)Genome modelling anddesign across all…Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026)
  • Rank Pangolin higher (4): Benchmarking splice variant prediction…, Amino-acid composition + fixed ridge…, MFASS v2 pinned rewire artifacts, A Reproducible MFASS Benchmark…
  • Rank SpliceAI higher (1): Genome modelling and design across all… (disagrees)
  • Mixed (3): Routine RNA-based analysis of potential…, Jaganathan et al. 2026, Cheng et al. 2026

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Pangolin or SpliceAI higherGenome modelling and design across all… (disagrees): Pangolin higher in 1, SpliceAI higher in 27, 0 ties. Routine RNA-based analysis of potential…: Pangolin higher in 6, SpliceAI higher in 6, 12 ties. Benchmarking splice variant prediction…: Pangolin higher in 8, SpliceAI higher in 4, 10 ties. Jaganathan et al. 2026: Pangolin higher in 6, SpliceAI higher in 8, 2 ties. Amino-acid composition + fixed ridge…: Pangolin higher in 4, SpliceAI higher in 0, 0 ties. MFASS v2 pinned rewire artifacts: Pangolin higher in 3, SpliceAI higher in 0, 0 ties. A Reproducible MFASS Benchmark…: Pangolin higher in 2, SpliceAI higher in 0, 0 ties. Cheng et al. 2026: Pangolin higher in 0, SpliceAI higher in 0, 1 tie.Disagrees: Genome modelling and design across all…Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026)1 Pangolin higher27 SpliceAI higher27Routine RNA-based analysis of potential…Routine RNA-based analysis of potential splicing variants facilitates genomic diagnostics and reveals limitations of in silico prediction tools6 Pangolin higher612 tie126 SpliceAI higher6Benchmarking splice variant prediction…Benchmarking splice variant prediction algorithms using massively parallel splicing assays8 Pangolin higher810 tie104 SpliceAI higher4Jaganathan et al. 2026A unified framework for quantitative splicing and transcript prediction (Jaganathan et al., bioRxiv 2026)6 Pangolin higher62 tie28 SpliceAI higher8Amino-acid composition + fixed ridge…Amino-acid composition + fixed ridge: automated execution audit4 Pangolin higher4MFASS v2 pinned rewire artifactsMFASS v2 pinned rewire artifacts3 Pangolin higher3A Reproducible MFASS Benchmark…A Reproducible MFASS Benchmark of Splice-Disruption Predictors Reveals a Shared Exon-Interior Blind Spot2 Pangolin higher2Cheng et al. 2026AlphaGenome Atlas: in silico mutagenesis of the entire human genome (Cheng, Taylor, Nicolaisen et al., medRxiv 2026)1 tie1
Pangolin higherTieSpliceAI higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Genome modelling and design across all… (28)
ProtocolDatasetMetricPangolinSpliceAIHigher
Zero-shot BRCA1 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUPRC (zero-shot; BRCA1 'Both' (coding and noncoding) SNVs), unitless0.290.508SpliceAI
Zero-shot BRCA1 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUROC (zero-shot; BRCA1 'Both' (coding and noncoding) SNVs), unitless0.5920.686SpliceAI
Zero-shot BRCA1 coding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUPRC (zero-shot; BRCA1 coding SNVs), unitless0.2490.284SpliceAI
Zero-shot BRCA1 coding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUROC (zero-shot; BRCA1 coding SNVs), unitless0.570.55Pangolin
Zero-shot BRCA1 intronic SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUPRC (zero-shot; BRCA1 intronic SNVs), unitless0.3480.357SpliceAI
Zero-shot BRCA1 intronic SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUROC (zero-shot; BRCA1 intronic SNVs), unitless0.8340.876SpliceAI
Zero-shot BRCA1 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUPRC (zero-shot; BRCA1 noncoding SNVs), unitless0.4220.862SpliceAI
Zero-shot BRCA1 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUROC (zero-shot; BRCA1 noncoding SNVs), unitless0.7410.965SpliceAI
Zero-shot BRCA1 splice region SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUPRC (zero-shot; BRCA1 splice region SNVs), unitless0.5560.901SpliceAI
Zero-shot BRCA1 splice region SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA1 saturation genome editing SNVs, Findlay et al. 2018 classes (Brixi et al. 2026)AUROC (zero-shot; BRCA1 splice region SNVs), unitless0.6750.947SpliceAI
Zero-shot BRCA2 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUPRC (zero-shot; BRCA2 'Both' (coding and noncoding) SNVs), unitless0.2990.393SpliceAI
Zero-shot BRCA2 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUROC (zero-shot; BRCA2 'Both' (coding and noncoding) SNVs), unitless0.5910.647SpliceAI
Zero-shot BRCA2 coding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUPRC (zero-shot; BRCA2 coding SNVs), unitless0.1790.208SpliceAI
Zero-shot BRCA2 coding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUROC (zero-shot; BRCA2 coding SNVs), unitless0.5450.56SpliceAI
Zero-shot BRCA2 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUPRC (zero-shot; BRCA2 noncoding SNVs), unitless0.7150.89SpliceAI
Zero-shot BRCA2 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)BRCA2 SNVs with functional classifications, Huang et al. 2025 (Brixi et al. 2026)AUROC (zero-shot; BRCA2 noncoding SNVs), unitless0.840.939SpliceAI
Zero-shot ClinVar noncoding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, highly conserved), unitless0.9810.998SpliceAI
Zero-shot ClinVar noncoding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, highly conserved), unitless0.880.987SpliceAI
Zero-shot ClinVar noncoding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, moderately conserved), unitless0.640.913SpliceAI
Zero-shot ClinVar noncoding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, moderately conserved), unitless0.8940.967SpliceAI
Zero-shot ClinVar noncoding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, not conserved), unitless0.6810.75SpliceAI
Zero-shot ClinVar noncoding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, not conserved), unitless0.9420.973SpliceAI
Zero-shot ClinVar noncoding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUPRC (zero-shot; ClinVar noncoding SNVs, poorly conserved), unitless0.6660.856SpliceAI
Zero-shot ClinVar noncoding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)ClinVar October 2024 release, filtered (Brixi et al. 2026)AUROC (zero-shot; ClinVar noncoding SNVs, poorly conserved), unitless0.9330.966SpliceAI
Zero-shot SpliceVarDB 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)SpliceVarDB experimentally validated splice variants (Brixi et al. 2026)AUPRC (zero-shot; SpliceVarDB 'Both' (coding and noncoding) SNVs), unitless0.9240.974SpliceAI
Zero-shot SpliceVarDB 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)SpliceVarDB experimentally validated splice variants (Brixi et al. 2026)AUROC (zero-shot; SpliceVarDB 'Both' (coding and noncoding) SNVs), unitless0.9030.961SpliceAI
Zero-shot SpliceVarDB exonic SNVs classification (Evo 2 paper, Supplementary Data 1)SpliceVarDB experimentally validated splice variants (Brixi et al. 2026)AUPRC (zero-shot; SpliceVarDB exonic SNVs), unitless0.8220.919SpliceAI
Zero-shot SpliceVarDB exonic SNVs classification (Evo 2 paper, Supplementary Data 1)SpliceVarDB experimentally validated splice variants (Brixi et al. 2026)AUROC (zero-shot; SpliceVarDB exonic SNVs), unitless0.8640.934SpliceAI
Comparisons in Routine RNA-based analysis of potential… (24)
ProtocolDatasetMetricPangolinSpliceAIHigher
Splice-effect prediction against patient-RNA or exon-trapping results, CAGI6 Splicing VUS subset (Drost et al. Data S1)CAGI6 Splicing VUS variants as scored by Drost et al.AUPRC (continuous score), fraction0.9580.956Tie
Splice-effect prediction against patient-RNA or exon-trapping results, CAGI6 Splicing VUS subset (Drost et al. Data S1)CAGI6 Splicing VUS variants as scored by Drost et al.AUROC (continuous score), fraction0.9130.909Tie
Splice-effect prediction against patient-RNA or exon-trapping results, CAGI6 Splicing VUS subset (Drost et al. Data S1)CAGI6 Splicing VUS variants as scored by Drost et al.F1 (binarised at each tool's literature threshold), fraction0.8890.889Tie
Splice-effect prediction against patient-RNA or exon-trapping results, CAGI6 Splicing VUS subset (Drost et al. Data S1)CAGI6 Splicing VUS variants as scored by Drost et al.Negative predictive value (binarised at each tool's literature threshold), fraction0.7620.762Tie
Splice-effect prediction against patient-RNA or exon-trapping results, CAGI6 Splicing VUS subset (Drost et al. Data S1)CAGI6 Splicing VUS variants as scored by Drost et al.Precision (binarised at each tool's literature threshold; column 'Pos Pred Value'), fraction0.9140.914Tie
Splice-effect prediction against patient-RNA or exon-trapping results, CAGI6 Splicing VUS subset (Drost et al. Data S1)CAGI6 Splicing VUS variants as scored by Drost et al.Precision (binarised at each tool's literature threshold; column 'Precision'), fraction0.9140.914Tie
Splice-effect prediction against patient-RNA or exon-trapping results, CAGI6 Splicing VUS subset (Drost et al. Data S1)CAGI6 Splicing VUS variants as scored by Drost et al.Recall (binarised at each tool's literature threshold; column 'Sensitivity (Recall)'), fraction0.8650.865Tie
Splice-effect prediction against patient-RNA or exon-trapping results, CAGI6 Splicing VUS subset (Drost et al. Data S1)CAGI6 Splicing VUS variants as scored by Drost et al.Specificity (binarised at each tool's literature threshold), fraction0.8420.842Tie
Splice-effect prediction against patient-RNA or exon-trapping results, in-house diagnostic cohort (Drost et al. Data S1)Drost et al. in-house RNA-tested diagnostic variants, scorable subsetAUPRC (continuous score), fraction0.9390.942Tie
Splice-effect prediction against patient-RNA or exon-trapping results, in-house diagnostic cohort (Drost et al. Data S1)Drost et al. in-house RNA-tested diagnostic variants, scorable subsetAUROC (continuous score), fraction0.8780.893SpliceAI
Splice-effect prediction against patient-RNA or exon-trapping results, in-house diagnostic cohort (Drost et al. Data S1)Drost et al. in-house RNA-tested diagnostic variants, scorable subsetF1 (binarised at each tool's literature threshold), fraction0.8560.858Tie
Splice-effect prediction against patient-RNA or exon-trapping results, in-house diagnostic cohort (Drost et al. Data S1)Drost et al. in-house RNA-tested diagnostic variants, scorable subsetNegative predictive value (binarised at each tool's literature threshold), fraction0.7540.769SpliceAI
Splice-effect prediction against patient-RNA or exon-trapping results, in-house diagnostic cohort (Drost et al. Data S1)Drost et al. in-house RNA-tested diagnostic variants, scorable subsetPrecision (binarised at each tool's literature threshold; column 'Pos Pred Value'), fraction0.8560.844Pangolin
Splice-effect prediction against patient-RNA or exon-trapping results, in-house diagnostic cohort (Drost et al. Data S1)Drost et al. in-house RNA-tested diagnostic variants, scorable subsetPrecision (binarised at each tool's literature threshold; column 'Precision'), fraction0.8560.844Pangolin
Splice-effect prediction against patient-RNA or exon-trapping results, in-house diagnostic cohort (Drost et al. Data S1)Drost et al. in-house RNA-tested diagnostic variants, scorable subsetRecall (binarised at each tool's literature threshold; column 'Sensitivity (Recall)'), fraction0.8560.873SpliceAI
Splice-effect prediction against patient-RNA or exon-trapping results, in-house diagnostic cohort (Drost et al. Data S1)Drost et al. in-house RNA-tested diagnostic variants, scorable subsetSpecificity (binarised at each tool's literature threshold), fraction0.7540.725Pangolin
Splice-effect prediction against patient-RNA or exon-trapping results, all 243 scorable variants (Drost et al. Data S1)Drost et al. RNA-tested clinical variants plus CAGI6 Splicing VUS, 243 scorableAUPRC (continuous score), fraction0.9430.943Tie
Splice-effect prediction against patient-RNA or exon-trapping results, all 243 scorable variants (Drost et al. Data S1)Drost et al. RNA-tested clinical variants plus CAGI6 Splicing VUS, 243 scorableAUROC (continuous score), fraction0.8850.895SpliceAI
Splice-effect prediction against patient-RNA or exon-trapping results, all 243 scorable variants (Drost et al. Data S1)Drost et al. RNA-tested clinical variants plus CAGI6 Splicing VUS, 243 scorableF1 (binarised at each tool's literature threshold), fraction0.8640.865Tie
Splice-effect prediction against patient-RNA or exon-trapping results, all 243 scorable variants (Drost et al. Data S1)Drost et al. RNA-tested clinical variants plus CAGI6 Splicing VUS, 243 scorableNegative predictive value (binarised at each tool's literature threshold), fraction0.7560.767SpliceAI
Splice-effect prediction against patient-RNA or exon-trapping results, all 243 scorable variants (Drost et al. Data S1)Drost et al. RNA-tested clinical variants plus CAGI6 Splicing VUS, 243 scorablePrecision (binarised at each tool's literature threshold; column 'Pos Pred Value'), fraction0.8690.86Pangolin
Splice-effect prediction against patient-RNA or exon-trapping results, all 243 scorable variants (Drost et al. Data S1)Drost et al. RNA-tested clinical variants plus CAGI6 Splicing VUS, 243 scorablePrecision (binarised at each tool's literature threshold; column 'Precision'), fraction0.8690.86Pangolin
Splice-effect prediction against patient-RNA or exon-trapping results, all 243 scorable variants (Drost et al. Data S1)Drost et al. RNA-tested clinical variants plus CAGI6 Splicing VUS, 243 scorableRecall (binarised at each tool's literature threshold; column 'Sensitivity (Recall)'), fraction0.8580.871SpliceAI
Splice-effect prediction against patient-RNA or exon-trapping results, all 243 scorable variants (Drost et al. Data S1)Drost et al. RNA-tested clinical variants plus CAGI6 Splicing VUS, 243 scorableSpecificity (binarised at each tool's literature threshold), fraction0.7730.75Pangolin
Comparisons in Benchmarking splice variant prediction… (22)
ProtocolDatasetMetricPangolinSpliceAIHigher
BRCA1 saturation genome editing, synonymous and intronic SNVs: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)BRCA1 saturation genome editing, synonymous and intronic SNVs (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; all variants), fraction0.9820.982Tie
BRCA1 saturation genome editing, synonymous and intronic SNVs: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)BRCA1 saturation genome editing, synonymous and intronic SNVs (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; exonic variants), fraction0.9090.909Tie
BRCA1 saturation genome editing, synonymous and intronic SNVs: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)BRCA1 saturation genome editing, synonymous and intronic SNVs (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; intronic variants), fraction0.9860.986Tie
BRCA1 saturation genome editing, synonymous and intronic SNVs: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)BRCA1 saturation genome editing, synonymous and intronic SNVs (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; intronic variants excluding essential splice sites), fraction0.9690.969Tie
FAS exon 6 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)FAS exon 6 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; all variants), fraction0.4870.539SpliceAI
FAS exon 6 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)FAS exon 6 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; exonic variants), fraction0.4870.539SpliceAI
MLH1 curated clinical splicing variants: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)MLH1 curated clinical splicing variants (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; all variants), fraction0.9310.931Tie
MLH1 curated clinical splicing variants: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)MLH1 curated clinical splicing variants (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; exonic variants), fraction0.7760.776Tie
MLH1 curated clinical splicing variants: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)MLH1 curated clinical splicing variants (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; intronic variants), fraction11Tie
MLH1 curated clinical splicing variants: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)MLH1 curated clinical splicing variants (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; intronic variants excluding essential splice sites), fraction11Tie
MST1R (RON) exon 11 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)MST1R (RON) exon 11 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; all variants), fraction0.330.237Pangolin
MST1R (RON) exon 11 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)MST1R (RON) exon 11 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; exonic variants), fraction0.3520.24Pangolin
MST1R (RON) exon 11 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)MST1R (RON) exon 11 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; intronic variants), fraction0.2890.232Pangolin
MST1R (RON) exon 11 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)MST1R (RON) exon 11 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; intronic variants excluding essential splice sites), fraction0.2350.174Pangolin
POU1F1 exon 2 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)POU1F1 exon 2 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; all variants), fraction0.9480.885Pangolin
POU1F1 exon 2 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)POU1F1 exon 2 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; exonic variants), fraction0.9430.9Pangolin
POU1F1 exon 2 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)POU1F1 exon 2 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; intronic variants), fraction0.9620.846Pangolin
POU1F1 exon 2 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)POU1F1 exon 2 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; intronic variants excluding essential splice sites), fraction0.950.8Pangolin
WT1 exon 9 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)WT1 exon 9 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; all variants), fraction0.6670.86SpliceAI
WT1 exon 9 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)WT1 exon 9 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; exonic variants), fraction0.360.8SpliceAI
WT1 exon 9 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)WT1 exon 9 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; intronic variants), fraction0.9060.906Tie
WT1 exon 9 saturation MPSA: transcriptome-normalised sensitivity at a 10% background call rate (Smith and Kitzman Table S2)WT1 exon 9 saturation MPSA (Smith and Kitzman benchmark set)Recall (transcriptome-normalised threshold calling 10% of background SNVs; intronic variants excluding essential splice sites), fraction0.850.85Tie
Comparisons in Jaganathan et al. 2026 (16)
ProtocolDatasetMetricPangolinSpliceAIHigher
Cryptic splice site activation by rare GT/AG-creating variants in Genomics England blood RNA-seq (SpliceAI2 paper, auPRC)Genomics England rare-disease participants with WGS and blood RNA-seq (Jaganathan et al. 2026)AUPRC (all variants), unitless0.40.48SpliceAI
Cryptic splice site activation by rare GT/AG-creating SNVs, GTEx WGS and RNA-seq (SpliceAI2 paper, auPRC)GTEx rare GT/AG-creating SNVs with RNA-seq cryptic splicing labels (Jaganathan et al. 2026)AUPRC (all variants; essential splice variants excluded), unitless0.560.66SpliceAI
Cryptic splice site activation by rare GT/AG-creating SNVs, GTEx WGS and RNA-seq (SpliceAI2 paper, auPRC)GTEx rare GT/AG-creating SNVs with RNA-seq cryptic splicing labels (Jaganathan et al. 2026)AUPRC (deep intronic variants (500 bp or more from the nearest exon); essential splice variants excluded), unitless0.440.52SpliceAI
Cryptic splice site activation by rare GT/AG-creating SNVs, GTEx WGS and RNA-seq (SpliceAI2 paper, auPRC)GTEx rare GT/AG-creating SNVs with RNA-seq cryptic splicing labels (Jaganathan et al. 2026)AUPRC (exonic variants; essential splice variants excluded), unitless0.640.73SpliceAI
Cryptic splice site activation by rare GT/AG-creating SNVs, GTEx WGS and RNA-seq (SpliceAI2 paper, auPRC)GTEx rare GT/AG-creating SNVs with RNA-seq cryptic splicing labels (Jaganathan et al. 2026)AUPRC (near intronic variants (under 500 bp from the nearest exon); essential splice variants excluded), unitless0.670.7SpliceAI
Cryptic splice site usage among RNA-seq-confirmed GT/AG-creating SNVs, GTEx (SpliceAI2 paper, Spearman)GTEx rare GT/AG-creating SNVs with RNA-seq cryptic splicing labels (Jaganathan et al. 2026)Spearman (all variants; essential splice variants excluded), unitless0.470.45Pangolin
Cryptic splice site usage among RNA-seq-confirmed GT/AG-creating SNVs, GTEx (SpliceAI2 paper, Spearman)GTEx rare GT/AG-creating SNVs with RNA-seq cryptic splicing labels (Jaganathan et al. 2026)Spearman (deep intronic variants (500 bp or more from the nearest exon); essential splice variants excluded), unitless0.350.36SpliceAI
Cryptic splice site usage among RNA-seq-confirmed GT/AG-creating SNVs, GTEx (SpliceAI2 paper, Spearman)GTEx rare GT/AG-creating SNVs with RNA-seq cryptic splicing labels (Jaganathan et al. 2026)Spearman (exonic variants; essential splice variants excluded), unitless0.510.47Pangolin
Cryptic splice site usage among RNA-seq-confirmed GT/AG-creating SNVs, GTEx (SpliceAI2 paper, Spearman)GTEx rare GT/AG-creating SNVs with RNA-seq cryptic splicing labels (Jaganathan et al. 2026)Spearman (near intronic variants (under 500 bp from the nearest exon); essential splice variants excluded), unitless0.520.5Pangolin
GTEx v8 fine-mapped sQTL SNVs against non-prioritised SNVs near junction ends (SpliceAI2 paper, auROC)GTEx v8 sQTL fine-mapped and null SNVs near junction ends (Jaganathan et al. 2026)AUROC (all variants; essential splice variants excluded), unitless0.70.71SpliceAI
GTEx v8 fine-mapped sQTL SNVs against non-prioritised SNVs near junction ends (SpliceAI2 paper, auROC)GTEx v8 sQTL fine-mapped and null SNVs near junction ends (Jaganathan et al. 2026)AUROC (exonic variants; essential splice variants excluded), unitless0.680.68Tie
GTEx v8 fine-mapped sQTL SNVs against non-prioritised SNVs near junction ends (SpliceAI2 paper, auROC)GTEx v8 sQTL fine-mapped and null SNVs near junction ends (Jaganathan et al. 2026)AUROC (intronic variants; essential splice variants excluded), unitless0.720.74SpliceAI
GTEx v8 fine-mapped sQTL effect size prediction (SpliceAI2 paper, Spearman)GTEx v8 sQTL fine-mapped and null SNVs near junction ends (Jaganathan et al. 2026)Spearman (all variants; essential splice variants excluded), unitless0.660.65Pangolin
GTEx v8 fine-mapped sQTL effect size prediction (SpliceAI2 paper, Spearman)GTEx v8 sQTL fine-mapped and null SNVs near junction ends (Jaganathan et al. 2026)Spearman (exonic variants; essential splice variants excluded), unitless0.620.62Tie
GTEx v8 fine-mapped sQTL effect size prediction (SpliceAI2 paper, Spearman)GTEx v8 sQTL fine-mapped and null SNVs near junction ends (Jaganathan et al. 2026)Spearman (intronic variants; essential splice variants excluded), unitless0.740.73Pangolin
Carrier plasma protein levels for rare splice-region variants, UK Biobank (SpliceAI2 paper, Pearson)UK Biobank rare splice-region variants with carrier plasma protein levels (Jaganathan et al. 2026)Pearson (with mean carrier plasma protein z-score; more negative is better), unitless-0.475-0.463Pangolin
Comparisons in Amino-acid composition + fixed ridge… (4)
Comparisons in MFASS v2 pinned rewire artifacts (3)
ProtocolDatasetMetricPangolinSpliceAIHigher
MFASS v2MFASS v2 eligible assay cohortAUROC, unitless0.8760.806Pangolin
MFASS v2MFASS v2 eligible assay cohortAverage precision, fraction0.3890.299Pangolin
MFASS v2MFASS v2 eligible assay cohortPrecision at 100, fraction0.650.64Pangolin
Comparisons in A Reproducible MFASS Benchmark… (2)
Comparisons in Cheng et al. 2026 (1)

What disagrees

Other studies
rank Pangolin above SpliceAI in 9 of 9 comparisons
This study
ranks SpliceAI above Pangolin in 27 of 28 comparisons
Compared
Pangolin, SpliceAI
Study that disagrees
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026), Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026)
Other studies
28 studiesAmino-acid composition + fixed ridge: automated execution audit, Training-mean control: automated execution audit, Sequence composition + RidgeCV: automated execution audit, Training-mean control: automated execution audit, ESM-2 8M masked-marginal scoring: automated execution audit, Amino-acid composition + fixed ridge: local execution report (20 September 2026), Training-mean control: local execution report (20 September 2026), Sequence composition + RidgeCV: local execution report (20 September 2026), Training-mean control: local execution report (20 September 2026), ESM-2 8M masked-marginal scoring: local execution report (20 September 2026), Protein composition + fixed ridge probe: audit, ESM-2 35M frozen residue-mean embeddings + fixed ridge (Rewire): audit, ESM-2 8M frozen residue-mean embeddings + fixed ridge (Rewire): audit, Training class prior / majority: audit, Seeded random ranking control: audit, Protein composition + fixed ridge probe: report, ESM-2 35M frozen residue-mean embeddings + fixed ridge (Rewire): report, ESM-2 8M frozen residue-mean embeddings + fixed ridge (Rewire): report, Training class prior / majority: report, Seeded random ranking control: report, MFASS matched canonical annotation v1: exclusion-verification.json, MFASS matched canonical annotation v1: manifest-v1.json, MFASS matched canonical annotation v1: provenance.json, MFASS matched canonical annotation v1: report.json, MFASS matched canonical annotation v1: verification.json, MFASS v2 corrected k-mer baseline: pinned upstream result JSON, MFASS v2 pinned rewire artifacts, A Reproducible MFASS Benchmark of Splice-Disruption Predictors Reveals a Shared Exon-Interior Blind Spot
Rule
Ordering reversal; unknown
Model identity
Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (56)
StudyToolProtocol or benchmarkMetricValue
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA1 'Both' (coding and noncoding) SNVs)0.29
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA1 'Both' (coding and noncoding) SNVs)0.592
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 coding SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA1 coding SNVs)0.249
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 coding SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA1 coding SNVs)0.57
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 intronic SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA1 intronic SNVs)0.348
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 intronic SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA1 intronic SNVs)0.834
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA1 noncoding SNVs)0.422
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA1 noncoding SNVs)0.741
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 splice region SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA1 splice region SNVs)0.556
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 splice region SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA1 splice region SNVs)0.675
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA2 'Both' (coding and noncoding) SNVs)0.299
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA2 'Both' (coding and noncoding) SNVs)0.591
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 coding SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA2 coding SNVs)0.179
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 coding SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA2 coding SNVs)0.545
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA2 noncoding SNVs)0.715
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA2 noncoding SNVs)0.84
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; ClinVar noncoding SNVs, highly conserved)0.981
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; ClinVar noncoding SNVs, highly conserved)0.88
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; ClinVar noncoding SNVs, moderately conserved)0.64
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; ClinVar noncoding SNVs, moderately conserved)0.894
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; ClinVar noncoding SNVs, not conserved)0.681
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; ClinVar noncoding SNVs, not conserved)0.942
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; ClinVar noncoding SNVs, poorly conserved)0.666
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; ClinVar noncoding SNVs, poorly conserved)0.933
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot SpliceVarDB 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; SpliceVarDB 'Both' (coding and noncoding) SNVs)0.924
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot SpliceVarDB 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; SpliceVarDB 'Both' (coding and noncoding) SNVs)0.903
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot SpliceVarDB exonic SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; SpliceVarDB exonic SNVs)0.822
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLPangolin (Evo 2 paper variant effect evaluations)Zero-shot SpliceVarDB exonic SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; SpliceVarDB exonic SNVs)0.864
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA1 'Both' (coding and noncoding) SNVs)0.508
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA1 'Both' (coding and noncoding) SNVs)0.686
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 coding SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA1 coding SNVs)0.284
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 coding SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA1 coding SNVs)0.55
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 intronic SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA1 intronic SNVs)0.357
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 intronic SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA1 intronic SNVs)0.876
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA1 noncoding SNVs)0.862
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA1 noncoding SNVs)0.965
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 splice region SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA1 splice region SNVs)0.901
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA1 splice region SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA1 splice region SNVs)0.947
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA2 'Both' (coding and noncoding) SNVs)0.393
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA2 'Both' (coding and noncoding) SNVs)0.647
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 coding SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA2 coding SNVs)0.208
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 coding SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA2 coding SNVs)0.56
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; BRCA2 noncoding SNVs)0.89
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot BRCA2 noncoding SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; BRCA2 noncoding SNVs)0.939
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; ClinVar noncoding SNVs, highly conserved)0.998
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, highly conserved classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; ClinVar noncoding SNVs, highly conserved)0.987
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; ClinVar noncoding SNVs, moderately conserved)0.913
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, moderately conserved classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; ClinVar noncoding SNVs, moderately conserved)0.967
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; ClinVar noncoding SNVs, not conserved)0.75
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, not conserved classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; ClinVar noncoding SNVs, not conserved)0.973
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; ClinVar noncoding SNVs, poorly conserved)0.856
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot ClinVar noncoding SNVs, poorly conserved classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; ClinVar noncoding SNVs, poorly conserved)0.966
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot SpliceVarDB 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; SpliceVarDB 'Both' (coding and noncoding) SNVs)0.974
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot SpliceVarDB 'Both' (coding and noncoding) SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; SpliceVarDB 'Both' (coding and noncoding) SNVs)0.961
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot SpliceVarDB exonic SNVs classification (Evo 2 paper, Supplementary Data 1)AUPRC (zero-shot; SpliceVarDB exonic SNVs)0.919
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026); Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026); evo2: Journal full-text XMLSpliceAI (Evo 2 paper variant effect evaluations)Zero-shot SpliceVarDB exonic SNVs classification (Evo 2 paper, Supplementary Data 1)AUROC (zero-shot; SpliceVarDB exonic SNVs)0.934
In the other studies (18)
StudyToolProtocol or benchmarkMetricValue
MFASS matched canonical annotation v1: report.json; MFASS matched canonical annotation v1: manifest-v1.json; MFASS matched canonical annotation v1: verification.json; MFASS matched canonical annotation v1: provenance.json; MFASS matched canonical annotation v1: exclusion-verification.jsonPangolin 1.0.2 + per-gene masking patch · mask False (P0)MFASS: matched GENCODE 44 canonical annotationAUROC0.876
MFASS matched canonical annotation v1: report.json; MFASS matched canonical annotation v1: manifest-v1.json; MFASS matched canonical annotation v1: verification.json; MFASS matched canonical annotation v1: provenance.json; MFASS matched canonical annotation v1: exclusion-verification.jsonPangolin 1.0.2 + per-gene masking patch · mask False (P0)MFASS: matched GENCODE 44 canonical annotationAverage precision0.389
MFASS matched canonical annotation v1: report.json; MFASS matched canonical annotation v1: manifest-v1.json; MFASS matched canonical annotation v1: verification.json; MFASS matched canonical annotation v1: provenance.json; MFASS matched canonical annotation v1: exclusion-verification.jsonPangolin 1.0.2 + per-gene masking patch · mask False (P0)MFASS: matched GENCODE 44 canonical annotationPrecision at 1000.65
MFASS matched canonical annotation v1: report.json; MFASS matched canonical annotation v1: manifest-v1.json; MFASS matched canonical annotation v1: verification.json; MFASS matched canonical annotation v1: provenance.json; MFASS matched canonical annotation v1: exclusion-verification.jsonPangolin 1.0.2 + per-gene masking patch · mask False (P0)MFASS: matched GENCODE 44 canonical annotationRecall at 1000.207
MFASS matched canonical annotation v1: report.json; MFASS matched canonical annotation v1: manifest-v1.json; MFASS matched canonical annotation v1: verification.json; MFASS matched canonical annotation v1: provenance.json; MFASS matched canonical annotation v1: exclusion-verification.jsonSpliceAI 1.3.1 · mask 0 (S0)MFASS: matched GENCODE 44 canonical annotationAUROC0.804
MFASS matched canonical annotation v1: report.json; MFASS matched canonical annotation v1: manifest-v1.json; MFASS matched canonical annotation v1: verification.json; MFASS matched canonical annotation v1: provenance.json; MFASS matched canonical annotation v1: exclusion-verification.jsonSpliceAI 1.3.1 · mask 0 (S0)MFASS: matched GENCODE 44 canonical annotationAverage precision0.295
MFASS matched canonical annotation v1: report.json; MFASS matched canonical annotation v1: manifest-v1.json; MFASS matched canonical annotation v1: verification.json; MFASS matched canonical annotation v1: provenance.json; MFASS matched canonical annotation v1: exclusion-verification.jsonSpliceAI 1.3.1 · mask 0 (S0)MFASS: matched GENCODE 44 canonical annotationPrecision at 1000.63
MFASS matched canonical annotation v1: report.json; MFASS matched canonical annotation v1: manifest-v1.json; MFASS matched canonical annotation v1: verification.json; MFASS matched canonical annotation v1: provenance.json; MFASS matched canonical annotation v1: exclusion-verification.jsonSpliceAI 1.3.1 · mask 0 (S0)MFASS: matched GENCODE 44 canonical annotationRecall at 1000.201
MFASS v2 pinned rewire artifacts; rewirebench: MFASS guide; rewirebench: MFASS protocol codePangolin · mask=FalseMFASS v2AUROC0.876
MFASS v2 pinned rewire artifacts; rewirebench: MFASS guide; rewirebench: MFASS protocol codePangolin · mask=FalseMFASS v2Average precision0.389
MFASS v2 pinned rewire artifacts; rewirebench: MFASS guide; rewirebench: MFASS protocol codePangolin · mask=FalseMFASS v2Precision at 1000.65
MFASS v2 pinned rewire artifacts; rewirebench: MFASS guide; rewirebench: MFASS protocol codeSpliceAI 1.3.1MFASS v2AUROC0.806
MFASS v2 pinned rewire artifacts; rewirebench: MFASS guide; rewirebench: MFASS protocol codeSpliceAI 1.3.1MFASS v2Average precision0.299
MFASS v2 pinned rewire artifacts; rewirebench: MFASS guide; rewirebench: MFASS protocol codeSpliceAI 1.3.1MFASS v2Precision at 1000.64
A Reproducible MFASS Benchmark of Splice-Disruption Predictors Reveals a Shared Exon-Interior Blind SpotPangolin (Znabu et al. 2026)MFASS splice-disrupting variant detection, all 27,733 labelled SNVs (Znabu et al. 2026)Average precision0.421
A Reproducible MFASS Benchmark of Splice-Disruption Predictors Reveals a Shared Exon-Interior Blind SpotPangolin (Znabu et al. 2026)MFASS splice-disrupting variant detection, all 27,733 labelled SNVs (Znabu et al. 2026)AUROC0.888
A Reproducible MFASS Benchmark of Splice-Disruption Predictors Reveals a Shared Exon-Interior Blind SpotSpliceAI (Znabu et al. 2026)MFASS splice-disrupting variant detection, all 27,733 labelled SNVs (Znabu et al. 2026)Average precision0.321
A Reproducible MFASS Benchmark of Splice-Disruption Predictors Reveals a Shared Exon-Interior Blind SpotSpliceAI (Znabu et al. 2026)MFASS splice-disrupting variant detection, all 27,733 labelled SNVs (Znabu et al. 2026)AUROC0.819

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-6353f9a63b02e765 · All contested results · How findings are made and followed up