PhyloGPN and Caduceus: A Phylogenetic Approach to Genomic Language Modeling and GUANinE v1.1 reveals complementarity of supervised and genomic language models disagree
- Low severity
- Not yet reviewed
- Two-study conflict
- unknown
A Phylogenetic Approach to Genomic Language Modeling ranks PhyloGPN above Caduceus in 12 of 12 comparisons; GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks Caduceus above PhyloGPN in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
2 papers rank PhyloGPN above Caduceus in 12 of their 18 comparisons and Caduceus higher in 6. Of those papers, GUANinE v1.1 reveals complementarity… ranks Caduceus higher on its own (6 of 6).
- Rank PhyloGPN higher (1): A Phylogenetic Approach to Genomic…
- Rank Caduceus higher (1): GUANinE v1.1 reveals complementarity…
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks PhyloGPN higher | Ranks Caduceus higher | Ties |
|---|---|---|---|---|
| A Phylogenetic Approach to Genomic Language Modeling and 2 more records · DOI | Agrees | 12 | 0 | 0 |
| GUANinE v1.1 reveals complementarity of supervised and genomic language models · DOI | Agrees | 0 | 6 | 0 |
Comparisons in A Phylogenetic Approach to Genomic… (12)
| Protocol | Dataset | Metric | PhyloGPN | Caduceus | Higher |
|---|---|---|---|---|---|
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Downstream of Transcript variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Downstream of Transcript), unitless | 0.97 | 0.72 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Intronic variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Intronic), unitless | 0.95 | 0.64 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Missense variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Missense), unitless | 0.84 | 0.55 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Noncoding variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Noncoding), unitless | 0.94 | 0.59 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Splice Acceptor variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Acceptor), unitless | 0.82 | 0.61 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Splice Donor variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Donor), unitless | 0.83 | 0.68 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Start Codon variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Start Codon), unitless | 0.77 | 0.41 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Stop Lost variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Stop Lost), unitless | 0.81 | 0.43 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Synonymous variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Synonymous), unitless | 0.85 | 0.55 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar Upstream of Transcript variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Upstream of Transcript), unitless | 0.74 | 0.4 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar 3’ UTR variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 3’ UTR), unitless | 0.94 | 0.63 | PhyloGPN |
| ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant class | ClinVar 5’ UTR variants, at least one review star | AUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 5’ UTR), unitless | 0.94 | 0.61 | PhyloGPN |
Comparisons in GUANinE v1.1 reveals complementarity… (6)
| Protocol | Dataset | Metric | PhyloGPN | Caduceus | Higher |
|---|---|---|---|---|---|
| GUANinE v1.1 ccre-propensity, Spearman rho x100 (Table 2) | GUANinE ccre-propensity test set | Spearman (GUANinE v1.1 test set), unitless | 18.4 | 41.3 | Caduceus |
| GUANinE v1.1 cons100, Spearman rho x100 (Table 2) | GUANinE cons100 test set | Spearman (GUANinE v1.1 test set), unitless | 43.1 | 62.4 | Caduceus |
| GUANinE v1.1 cons30, Spearman rho x100 (Table 2) | GUANinE cons30 test set | Spearman (GUANinE v1.1 test set), unitless | 53.2 | 74.4 | Caduceus |
| GUANinE v1.1 dnase-propensity, Spearman rho x100 (Table 2) | GUANinE dnase-propensity test set | Spearman (GUANinE v1.1 test set), unitless | 40.5 | 43 | Caduceus |
| GUANinE v1.1 gpra-c, Spearman rho x100 (Table 2) | GUANinE gpra-c test set | Spearman (GUANinE v1.1 test set), unitless | 66.9 | 72.4 | Caduceus |
| GUANinE v1.1 gpra-d, Spearman rho x100 (Table 2) | GUANinE gpra-d test set | Spearman (GUANinE v1.1 test set), unitless | 70.7 | 76 | Caduceus |
What disagrees
- studies
- 2
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-b0dc3ee811c0d87f · All contested results · How findings are made and followed up