ESM-2 and CADD: Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) reverses the order
- Low severity
- Not yet reviewed
- Ordering reversal
- Different tasks across studies
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) ranks ESM-2 above CADD in 6 of 7 comparisons; other studies rank CADD above ESM-2 in 14 of 14.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
Genome modelling and design across all… ranks CADD above ESM-2 in all 14 of its comparisons. Livesey and Marsh 2025 ranks ESM-2 above CADD in 6 of its 7 comparisons and CADD higher in 1.
- Rank CADD higher (1): Genome modelling and design across all…
- Rank ESM-2 higher (1): Livesey and Marsh 2025 (disagrees)
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks CADD higher | Ranks ESM-2 higher | Ties |
|---|---|---|---|---|
| Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) and 1 more record · DOI | Disagrees | 1 | 6 | 0 |
| Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026) and 1 more record · DOI | Agrees | 14 | 0 | 0 |
Comparisons in Livesey and Marsh 2025 (7)
| Protocol | Dataset | Metric | CADD | ESM-2 | Higher |
|---|---|---|---|---|---|
| Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense | Livesey and Marsh 2025 clinical set: ClinVar pathogenic versus gnomAD v4.1 missense (819 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, per-protein AUROC on shared variants, ClinVar P/LP versus gnomAD v4.1 missense, proteins with at least 10 of each), percent | 52.1 | 51.4 | CADD |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets), percent | 40.3 | 72.2 | ESM-2 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), |Spearman| with DMS on shared variants, 36 human DMS sets), percent | 2.89 | 27.8 | ESM-2 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, direct DMS assays only), percent | 39 | 74.2 | ESM-2 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, indirect DMS assays only), percent | 41.2 | 70.8 | ESM-2 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions only | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, SNV-accessible substitutions only), percent | 40.3 | 70.5 | ESM-2 |
| Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filled | Livesey and Marsh 2025 human DMS benchmark set (36 proteins) | Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score), percent | 42 | 71.1 | ESM-2 |
Comparisons in Genome modelling and design across all… (14)
What disagrees
- Other studies
- rank CADD above ESM-2 in 14 of 14 comparisons
- This study
- ranks ESM-2 above CADD in 6 of 7 comparisons
- Compared
- ESM-2, CADD
- Study that disagrees
- Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML), Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8)
- Other studies
- Genome modelling and design across all domains of life with Evo 2: Supplementary Data 1, human variant effect prediction results (Nature 2026), Genome modelling and design across all domains of life with Evo 2: Supplementary Information (Nature 2026)
- Rule
- Ordering reversal; different tasks across studies
- Model identity
- Matched across studies partly through links not yet reviewed
The compared numbers
Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-e9051c33e59061d8 · All contested results · How findings are made and followed up