Contested result

Evo 2 and Caduceus: GUANinE v1.1 reveals complementarity of supervised and genomic language models and GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models disagree

Low severity
Not yet reviewed
two-study-conflict
different-task

GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks Caduceus above Evo 2 in 4 of 4 comparisons; GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models ranks Evo 2 above Caduceus in 35 of 46 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

3 papers rank Evo 2 above Caduceus in 36 of their 58 comparisons and Caduceus higher in 15; 7 are ties. Of those papers, GUANinE v1.1 reveals complementarity… ranks Caduceus higher on its own (4 of 6).

Rank Evo 2 higher (2): GFMBench-API, TraitGym official metrics
Rank Caduceus higher (1): GUANinE v1.1 reveals complementarity…
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Evo 2 higherTieCaduceus higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (3)
Paper	Role	Ranks Evo 2 higher	Ranks Caduceus higher	Ties
GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models · DOI	Agrees	35	11	4
GUANinE v1.1 reveals complementarity of supervised and genomic language models · DOI	Agrees	0	4	2
TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym) · Source	Agrees	1	0	1

Comparisons in GFMBench-API (50)

Protocol	Dataset	Metric	Evo 2	Caduceus	Higher
GFMBench-API Bend Variant Effects Disease, zs-embedding	Bend Variant Effects Disease test set (GFMBench-API)	AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.596	0.379	Evo 2
GFMBench-API Bend Variant Effects Disease, zs-embedding	Bend Variant Effects Disease test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.0944	0.107	Caduceus
GFMBench-API Bend Variant Effects Disease, zs-embedding	Bend Variant Effects Disease test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.474	0.378	Evo 2
GFMBench-API Bend Variant Effects Disease, zs-embedding	Bend Variant Effects Disease test set (GFMBench-API)	AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.912	0.78	Evo 2
GFMBench-API Bend Variant Effects Disease, zs-embedding	Bend Variant Effects Disease test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.615	0.545	Evo 2
GFMBench-API Bend Variant Effects Disease, zs-embedding	Bend Variant Effects Disease test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.872	0.772	Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embedding	Bend Variant Effects Expression test set (GFMBench-API)	AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.0897	0.0692	Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embedding	Bend Variant Effects Expression test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.0887	0.0775	Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embedding	Bend Variant Effects Expression test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.0904	0.0676	Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embedding	Bend Variant Effects Expression test set (GFMBench-API)	AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.573	0.479	Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embedding	Bend Variant Effects Expression test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.557	0.537	Evo 2
GFMBench-API Bend Variant Effects Expression, zs-embedding	Bend Variant Effects Expression test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.582	0.473	Evo 2
GFMBench-API BRCA1, zs-embedding	BRCA1 test set (GFMBench-API)	AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.575	0.299	Evo 2
GFMBench-API BRCA1, zs-embedding	BRCA1 test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.204	0.203	Tie
GFMBench-API BRCA1, zs-embedding	BRCA1 test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.54	0.291	Evo 2
GFMBench-API BRCA1, zs-embedding	BRCA1 test set (GFMBench-API)	AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.775	0.563	Evo 2
GFMBench-API BRCA1, zs-embedding	BRCA1 test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.467	0.469	Tie
GFMBench-API BRCA1, zs-embedding	BRCA1 test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.747	0.555	Evo 2
GFMBench-API ClinVar Indel Pathogenicity, zs-embedding	ClinVar Indel Pathogenicity test set (GFMBench-API)	AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.993	0.959	Evo 2
GFMBench-API ClinVar Indel Pathogenicity, zs-embedding	ClinVar Indel Pathogenicity test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.989	0.958	Evo 2
GFMBench-API ClinVar Indel Pathogenicity, zs-embedding	ClinVar Indel Pathogenicity test set (GFMBench-API)	AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.967	0.71	Evo 2
GFMBench-API ClinVar Indel Pathogenicity, zs-embedding	ClinVar Indel Pathogenicity test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.946	0.708	Evo 2
GFMBench-API LOL-EVE, zs-embedding	LOL-EVE test set (GFMBench-API)	AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.0938	0.0863	Evo 2
GFMBench-API LOL-EVE, zs-embedding	LOL-EVE test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.0845	0.0861	Caduceus
GFMBench-API LOL-EVE, zs-embedding	LOL-EVE test set (GFMBench-API)	AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.543	0.522	Evo 2
GFMBench-API LOL-EVE, zs-embedding	LOL-EVE test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.531	0.524	Evo 2
GFMBench-API LRB Variant Effect OMIM, zs-embedding	LRB Variant Effect OMIM test set (GFMBench-API)	AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.0021	0.0019	Evo 2
GFMBench-API LRB Variant Effect OMIM, zs-embedding	LRB Variant Effect OMIM test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.002	0.0022	Caduceus
GFMBench-API LRB Variant Effect OMIM, zs-embedding	LRB Variant Effect OMIM test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.002	0.0019	Evo 2
GFMBench-API LRB Variant Effect OMIM, zs-embedding	LRB Variant Effect OMIM test set (GFMBench-API)	AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.502	0.49	Evo 2
GFMBench-API LRB Variant Effect OMIM, zs-embedding	LRB Variant Effect OMIM test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.498	0.524	Caduceus
GFMBench-API LRB Variant Effect OMIM, zs-embedding	LRB Variant Effect OMIM test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.501	0.486	Evo 2
GFMBench-API Songlab ClinVar, zs-embedding	Songlab ClinVar test set (GFMBench-API)	AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.865	0.991	Caduceus
GFMBench-API Songlab ClinVar, zs-embedding	Songlab ClinVar test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.576	0.991	Caduceus
GFMBench-API Songlab ClinVar, zs-embedding	Songlab ClinVar test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.734	0.996	Caduceus
GFMBench-API Songlab ClinVar, zs-embedding	Songlab ClinVar test set (GFMBench-API)	AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.845	0.867	Caduceus
GFMBench-API Songlab ClinVar, zs-embedding	Songlab ClinVar test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.579	0.867	Caduceus
GFMBench-API Songlab ClinVar, zs-embedding	Songlab ClinVar test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.722	0.933	Caduceus
GFMBench-API TraitGym Complex, zs-embedding	TraitGym Complex test set (GFMBench-API)	AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.107	0.0974	Evo 2
GFMBench-API TraitGym Complex, zs-embedding	TraitGym Complex test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.106	0.103	Evo 2
GFMBench-API TraitGym Complex, zs-embedding	TraitGym Complex test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.108	0.0946	Evo 2
GFMBench-API TraitGym Complex, zs-embedding	TraitGym Complex test set (GFMBench-API)	AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.524	0.497	Evo 2
GFMBench-API TraitGym Complex, zs-embedding	TraitGym Complex test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.523	0.518	Tie
GFMBench-API TraitGym Complex, zs-embedding	TraitGym Complex test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.527	0.483	Evo 2
GFMBench-API TraitGym Mendelian, zs-embedding	TraitGym Mendelian test set (GFMBench-API)	AUPRC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.17	0.0922	Evo 2
GFMBench-API TraitGym Mendelian, zs-embedding	TraitGym Mendelian test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.121	0.127	Caduceus
GFMBench-API TraitGym Mendelian, zs-embedding	TraitGym Mendelian test set (GFMBench-API)	AUPRC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.164	0.0879	Evo 2
GFMBench-API TraitGym Mendelian, zs-embedding	TraitGym Mendelian test set (GFMBench-API)	AUROC (zero-shot; L2 distance of reference and alternate sequence embeddings), unitless	0.626	0.477	Evo 2
GFMBench-API TraitGym Mendelian, zs-embedding	TraitGym Mendelian test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of embeddings at the SNV position), unitless	0.548	0.553	Tie
GFMBench-API TraitGym Mendelian, zs-embedding	TraitGym Mendelian test set (GFMBench-API)	AUROC (zero-shot; cosine similarity of reference and alternate sequence embeddings), unitless	0.618	0.463	Evo 2

Comparisons in GUANinE v1.1 reveals complementarity… (6)

Protocol	Dataset	Metric	Evo 2	Caduceus	Higher
GUANinE v1.1 ccre-propensity, Spearman rho x100 (Table 2)	GUANinE ccre-propensity test set	Spearman (GUANinE v1.1 test set), unitless	40.2	41.3	Caduceus
GUANinE v1.1 cons100, Spearman rho x100 (Table 2)	GUANinE cons100 test set	Spearman (GUANinE v1.1 test set), unitless	55.2	62.4	Caduceus
GUANinE v1.1 cons30, Spearman rho x100 (Table 2)	GUANinE cons30 test set	Spearman (GUANinE v1.1 test set), unitless	68.5	74.4	Caduceus
GUANinE v1.1 dnase-propensity, Spearman rho x100 (Table 2)	GUANinE dnase-propensity test set	Spearman (GUANinE v1.1 test set), unitless	41.8	43	Caduceus
GUANinE v1.1 gpra-c, Spearman rho x100 (Table 2)	GUANinE gpra-c test set	Spearman (GUANinE v1.1 test set), unitless	72.7	72.4	Tie
GUANinE v1.1 gpra-d, Spearman rho x100 (Table 2)	GUANinE gpra-d test set	Spearman (GUANinE v1.1 test set), unitless	76.4	76	Tie

Comparisons in TraitGym official metrics (2)

Protocol	Dataset	Metric	Evo 2	Caduceus	Higher
TraitGym Complex traits, zero-shot, AUPRC by chromosome weighted average	TraitGym Complex traits, matched 9 controls	AUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all'), unitless	0.0976	0.0971	Tie
TraitGym Mendelian traits, zero-shot, AUPRC by chromosome weighted average	TraitGym Mendelian traits, matched 9 controls	AUPRC (weighted average over chromosomes of per-chromosome AUPRC; subset 'all'), unitless	0.159	0.0878	Evo 2

What disagrees

studies: 2

Compared: Evo 2, Caduceus
Study that disagrees
Other studies
Rule: two-study-conflict; different-task
Model identity: Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-5b81076ca7ef0584 · All contested results · How findings are made and followed up