rewirebio.iobenchmarks
Contested result

DNABERT-2 and PhyloGPN: GUANinE v1.1 reveals complementarity of supervised and genomic language models and Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) disagree

  • Low severity
  • Not yet reviewed
  • two-study-conflict
  • unknown

GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks DNABERT-2 above PhyloGPN in 6 of 6 comparisons; Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) ranks PhyloGPN above DNABERT-2 in 14 of 14 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank DNABERT-2 above PhyloGPN in 6 of their 20 comparisons and PhyloGPN higher in 14. Of those papers, Genomic heterogeneity inflates… ranks PhyloGPN higher on its own (14 of 14).

Papers grouped by which system they rank higher: DNABERT-2 or PhyloGPNRank DNABERT-2 higher: GUANinE v1.1 reveals complementarity…. Rank PhyloGPN higher: Genomic heterogeneity inflates…. Mixed: none.Rank DNABERT-2 higherMixedRank PhyloGPN higherGUANinE v1.1 revealscomplementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language modelsGenomic heterogeneityinflates…Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)
  • Rank DNABERT-2 higher (1): GUANinE v1.1 reveals complementarity…
  • Rank PhyloGPN higher (1): Genomic heterogeneity inflates…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores DNABERT-2 or PhyloGPN higherGenomic heterogeneity inflates…: DNABERT-2 higher in 0, PhyloGPN higher in 14, 0 ties. GUANinE v1.1 reveals complementarity…: DNABERT-2 higher in 6, PhyloGPN higher in 0, 0 ties.Genomic heterogeneity inflates…Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2)14 PhyloGPN higher14GUANinE v1.1 reveals complementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language models6 DNABERT-2 higher6
DNABERT-2 higherTiePhyloGPN higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks DNABERT-2 higherRanks PhyloGPN higherTies
Genomic heterogeneity inflates the performance of variant pathogenicity predictions (Lu, Liu, Lin and Brandes, bioRxiv 2025, v2) · DOIAgrees0140
GUANinE v1.1 reveals complementarity of supervised and genomic language models · DOIAgrees600
Comparisons in Genomic heterogeneity inflates… (14)
ProtocolDatasetMetricDNABERT-2PhyloGPNHigher
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: All variants (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, All variants), unitless0.5760.915PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Intron (Non-splice) (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Intron (Non-splice)), unitless0.4370.846PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Missense (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Missense), unitless0.4840.799PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Missense & Intron (Non-splice) (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Missense & Intron (Non-splice)), unitless0.4490.689PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Missense & 3'UTR (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Missense & 3'UTR), unitless0.5460.729PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Noncoding (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Noncoding), unitless0.4330.951PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: RNA gene (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, RNA gene), unitless0.4230.476PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Splice (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Splice), unitless0.4540.722PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Start loss (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Start loss), unitless0.3890.583PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Stop gain (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Stop gain), unitless0.4830.677PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Stop loss (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Stop loss), unitless0.570.695PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: Synonymous (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, Synonymous), unitless0.5520.806PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: 3' UTR (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, 3' UTR), unitless0.4990.765PhyloGPN
ClinVar pathogenic versus benign SNVs within variant type, AUROC (Lu et al. 2025)Lu et al. 2025 ClinVar benchmark: 5' UTR (Figure 1)AUROC (ClinVar pathogenic vs benign SNVs, 5' UTR), unitless0.4140.601PhyloGPN
Comparisons in GUANinE v1.1 reveals complementarity… (6)
ProtocolDatasetMetricDNABERT-2PhyloGPNHigher
GUANinE v1.1 ccre-propensity, Spearman rho x100 (Table 2)GUANinE ccre-propensity test setSpearman (GUANinE v1.1 test set), unitless32.818.4DNABERT-2
GUANinE v1.1 cons100, Spearman rho x100 (Table 2)GUANinE cons100 test setSpearman (GUANinE v1.1 test set), unitless50.443.1DNABERT-2
GUANinE v1.1 cons30, Spearman rho x100 (Table 2)GUANinE cons30 test setSpearman (GUANinE v1.1 test set), unitless60.653.2DNABERT-2
GUANinE v1.1 dnase-propensity, Spearman rho x100 (Table 2)GUANinE dnase-propensity test setSpearman (GUANinE v1.1 test set), unitless41.740.5DNABERT-2
GUANinE v1.1 gpra-c, Spearman rho x100 (Table 2)GUANinE gpra-c test setSpearman (GUANinE v1.1 test set), unitless69.966.9DNABERT-2
GUANinE v1.1 gpra-d, Spearman rho x100 (Table 2)GUANinE gpra-d test setSpearman (GUANinE v1.1 test set), unitless73.970.7DNABERT-2

What disagrees

studies
2
Compared
DNABERT-2, PhyloGPN
Study that disagrees
Other studies
Rule
two-study-conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-17a4ccabb79c397b · All contested results · How findings are made and followed up