Pangolin and SpliceAI: Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) reverses the order
- Low severity
- Not yet reviewed
- Ordering reversal
- unknown
Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) ranks SpliceAI above Pangolin in 27 of 28 comparisons; 3 of 3 other studies rank Pangolin above SpliceAI.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
7 papers rank Pangolin above SpliceAI in 29 of their 72 comparisons and SpliceAI higher in 18; 25 are ties. Genome modelling and design across all… ranks SpliceAI above Pangolin in 27 of its 28 comparisons and Pangolin higher in 1.
- Rank Pangolin higher (4): Benchmarking splice variant prediction…, Amino-acid composition + fixed ridge…, MFASS v2 pinned rewire artifacts, A Reproducible MFASS Benchmark…
- Rank SpliceAI higher (1): Genome modelling and design across all… (disagrees)
- Mixed (3): Routine RNA-based analysis of potential…, Jaganathan et al. 2026, Cheng et al. 2026
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
Comparisons in Genome modelling and design across all… (28)
Comparisons in Routine RNA-based analysis of potential… (24)
Comparisons in Benchmarking splice variant prediction… (22)
Comparisons in Jaganathan et al. 2026 (16)
Comparisons in Amino-acid composition + fixed ridge… (4)
| Protocol | Dataset | Metric | Pangolin | SpliceAI | Higher |
|---|---|---|---|---|---|
| MFASS: matched GENCODE 44 canonical annotation | MFASS v2 test: matched canonical annotation coverage | AUROC, unitless | 0.876 | 0.804 | Pangolin |
| MFASS: matched GENCODE 44 canonical annotation | MFASS v2 test: matched canonical annotation coverage | Average precision, unitless | 0.389 | 0.295 | Pangolin |
| MFASS: matched GENCODE 44 canonical annotation | MFASS v2 test: matched canonical annotation coverage | Precision at 100, unitless | 0.65 | 0.63 | Pangolin |
| MFASS: matched GENCODE 44 canonical annotation | MFASS v2 test: matched canonical annotation coverage | Recall at 100, unitless | 0.207 | 0.201 | Pangolin |
Comparisons in MFASS v2 pinned rewire artifacts (3)
| Protocol | Dataset | Metric | Pangolin | SpliceAI | Higher |
|---|---|---|---|---|---|
| MFASS v2 | MFASS v2 eligible assay cohort | AUROC, unitless | 0.876 | 0.806 | Pangolin |
| MFASS v2 | MFASS v2 eligible assay cohort | Average precision, fraction | 0.389 | 0.299 | Pangolin |
| MFASS v2 | MFASS v2 eligible assay cohort | Precision at 100, fraction | 0.65 | 0.64 | Pangolin |
Comparisons in A Reproducible MFASS Benchmark… (2)
| Protocol | Dataset | Metric | Pangolin | SpliceAI | Higher |
|---|---|---|---|---|---|
| MFASS splice-disrupting variant detection, all 27,733 labelled SNVs (Znabu et al. 2026) | MFASS v2 eligible assay cohort | AUROC, unitless | 0.888 | 0.819 | Pangolin |
| MFASS splice-disrupting variant detection, all 27,733 labelled SNVs (Znabu et al. 2026) | MFASS v2 eligible assay cohort | Average precision, unitless | 0.421 | 0.321 | Pangolin |
Comparisons in Cheng et al. 2026 (1)
| Protocol | Dataset | Metric | Pangolin | SpliceAI | Higher |
|---|---|---|---|---|---|
| DNM1 minigene screen, active versus inactive variants, average precision (AlphaGenome Atlas) | DNM1 intron saturation mutagenesis minigene splicing screen (AlphaGenome Atlas Data S1) | Average precision (DNM1 minigene screen, active vs inactive variants), unitless | 0.939 | 0.94 | Tie |
What disagrees
- Other studies
- rank Pangolin above SpliceAI in 9 of 9 comparisons
- This study
- ranks SpliceAI above Pangolin in 27 of 28 comparisons
- Compared
- Pangolin, SpliceAI
- Study that disagrees
- Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026), Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026)
- Other studies
28 studies
Amino-acid composition + fixed ridge: automated execution audit, Training-mean control: automated execution audit, Sequence composition + RidgeCV: automated execution audit, Training-mean control: automated execution audit, ESM-2 8M masked-marginal scoring: automated execution audit, Amino-acid composition + fixed ridge: local execution report (20 September 2026), Training-mean control: local execution report (20 September 2026), Sequence composition + RidgeCV: local execution report (20 September 2026), Training-mean control: local execution report (20 September 2026), ESM-2 8M masked-marginal scoring: local execution report (20 September 2026), Protein composition + fixed ridge probe: audit, ESM-2 35M frozen residue-mean embeddings + fixed ridge (Rewire): audit, ESM-2 8M frozen residue-mean embeddings + fixed ridge (Rewire): audit, Training class prior / majority: audit, Seeded random ranking control: audit, Protein composition + fixed ridge probe: report, ESM-2 35M frozen residue-mean embeddings + fixed ridge (Rewire): report, ESM-2 8M frozen residue-mean embeddings + fixed ridge (Rewire): report, Training class prior / majority: report, Seeded random ranking control: report, MFASS matched canonical annotation v1: exclusion-verification.json, MFASS matched canonical annotation v1: manifest-v1.json, MFASS matched canonical annotation v1: provenance.json, MFASS matched canonical annotation v1: report.json, MFASS matched canonical annotation v1: verification.json, MFASS v2 corrected k-mer baseline: pinned upstream result JSON, MFASS v2 pinned rewire artifacts, A Reproducible MFASS Benchmark of Splice-Disruption Predictors Reveals a Shared Exon-Interior Blind Spot- Rule
- Ordering reversal; unknown
- Model identity
- Matched across studies partly through links not yet reviewed
The compared numbers
Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-6353f9a63b02e765 · All contested results · How findings are made and followed up