rewirebio.iobenchmarks
Contested result

PhyloGPN and Caduceus: A Phylogenetic Approach to Genomic Language Modeling and GUANinE v1.1 reveals complementarity of supervised and genomic language models disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

A Phylogenetic Approach to Genomic Language Modeling ranks PhyloGPN above Caduceus in 12 of 12 comparisons; GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks Caduceus above PhyloGPN in 6 of 6 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank PhyloGPN above Caduceus in 12 of their 18 comparisons and Caduceus higher in 6. Of those papers, GUANinE v1.1 reveals complementarity… ranks Caduceus higher on its own (6 of 6).

Papers grouped by which system they rank higher: PhyloGPN or CaduceusRank PhyloGPN higher: A Phylogenetic Approach to Genomic…. Rank Caduceus higher: GUANinE v1.1 reveals complementarity…. Mixed: none.Rank PhyloGPN higherMixedRank Caduceus higherA PhylogeneticApproach to Genomic…A Phylogenetic Approach to Genomic Language ModelingGUANinE v1.1 revealscomplementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language models
  • Rank PhyloGPN higher (1): A Phylogenetic Approach to Genomic…
  • Rank Caduceus higher (1): GUANinE v1.1 reveals complementarity…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores PhyloGPN or Caduceus higherA Phylogenetic Approach to Genomic…: PhyloGPN higher in 12, Caduceus higher in 0, 0 ties. GUANinE v1.1 reveals complementarity…: PhyloGPN higher in 0, Caduceus higher in 6, 0 ties.A Phylogenetic Approach to Genomic…A Phylogenetic Approach to Genomic Language Modeling12 PhyloGPN higher12GUANinE v1.1 reveals complementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language models6 Caduceus higher6
PhyloGPN higherTieCaduceus higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks PhyloGPN higherRanks Caduceus higherTies
A Phylogenetic Approach to Genomic Language Modeling and 2 more records · DOIAgrees1200
GUANinE v1.1 reveals complementarity of supervised and genomic language models · DOIAgrees060
Comparisons in A Phylogenetic Approach to Genomic… (12)
ProtocolDatasetMetricPhyloGPNCaduceusHigher
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Downstream of Transcript variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Downstream of Transcript), unitless0.970.72PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Intronic variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Intronic), unitless0.950.64PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Missense variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Missense), unitless0.840.55PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Noncoding variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Noncoding), unitless0.940.59PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Splice Acceptor variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Acceptor), unitless0.820.61PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Splice Donor variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Splice Donor), unitless0.830.68PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Start Codon variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Start Codon), unitless0.770.41PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Stop Lost variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Stop Lost), unitless0.810.43PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Synonymous variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Synonymous), unitless0.850.55PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar Upstream of Transcript variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class Upstream of Transcript), unitless0.740.4PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar 3’ UTR variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 3’ UTR), unitless0.940.63PhyloGPN
ClinVar pathogenic versus benign classification from likelihood ratios, AUROC by variant classClinVar 5’ UTR variants, at least one review starAUROC (ClinVar pathogenic or likely pathogenic versus benign or likely benign, zero-shot allele log likelihood ratio, class 5’ UTR), unitless0.940.61PhyloGPN
Comparisons in GUANinE v1.1 reveals complementarity… (6)
ProtocolDatasetMetricPhyloGPNCaduceusHigher
GUANinE v1.1 ccre-propensity, Spearman rho x100 (Table 2)GUANinE ccre-propensity test setSpearman (GUANinE v1.1 test set), unitless18.441.3Caduceus
GUANinE v1.1 cons100, Spearman rho x100 (Table 2)GUANinE cons100 test setSpearman (GUANinE v1.1 test set), unitless43.162.4Caduceus
GUANinE v1.1 cons30, Spearman rho x100 (Table 2)GUANinE cons30 test setSpearman (GUANinE v1.1 test set), unitless53.274.4Caduceus
GUANinE v1.1 dnase-propensity, Spearman rho x100 (Table 2)GUANinE dnase-propensity test setSpearman (GUANinE v1.1 test set), unitless40.543Caduceus
GUANinE v1.1 gpra-c, Spearman rho x100 (Table 2)GUANinE gpra-c test setSpearman (GUANinE v1.1 test set), unitless66.972.4Caduceus
GUANinE v1.1 gpra-d, Spearman rho x100 (Table 2)GUANinE gpra-d test setSpearman (GUANinE v1.1 test set), unitless70.776Caduceus

What disagrees

studies
2
Compared
PhyloGPN, Caduceus
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-b0dc3ee811c0d87f · All contested results · How findings are made and followed up