Evo 2 and Mistral-DNA: GUANinE v1.1 reveals complementarity of supervised and genomic language models and BacBench LICENSE file at commit 64b7d45 disagree
- Low severity
- Not yet reviewed
- Two-study conflict
- different-task
GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks Evo 2 above Mistral-DNA in 6 of 6 comparisons; BacBench LICENSE file at commit 64b7d45 ranks Mistral-DNA above Evo 2 in 9 of 10 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
2 papers rank Evo 2 above Mistral-DNA in 7 of their 16 comparisons and Mistral-DNA higher in 9. Of those papers, BacBench LICENSE file at commit 64b7d45 ranks Mistral-DNA higher on its own (9 of 10).
- Rank Evo 2 higher (1): GUANinE v1.1 reveals complementarity…
- Rank Mistral-DNA higher (1): BacBench LICENSE file at commit 64b7d45
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks Evo 2 higher | Ranks Mistral-DNA higher | Ties |
|---|---|---|---|---|
| BacBench LICENSE file at commit 64b7d45 and 12 more records · DOI | Agrees | 1 | 9 | 0 |
| GUANinE v1.1 reveals complementarity of supervised and genomic language models · DOI | Agrees | 6 | 0 | 0 |
Comparisons in BacBench LICENSE file at commit 64b7d45 (10)
| Protocol | Dataset | Metric | Evo 2 | Mistral-DNA | Higher |
|---|---|---|---|---|---|
| BacBench antibiotic resistance, binary linear probe AUPRC | Antibiotic resistance binary prediction subset, 37 antibiotics | AUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless | 38.4 | 49.9 | Mistral-DNA |
| BacBench antibiotic resistance, binary linear probe AUROC | Antibiotic resistance binary prediction subset, 37 antibiotics | AUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless | 66.3 | 76 | Mistral-DNA |
| BacBench antibiotic resistance, MIC regression Pearson correlation | Antibiotic resistance MIC regression subset, 56 antibiotics | Pearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless | 0.53 | 41 | Mistral-DNA |
| BacBench antibiotic resistance, MIC regression R squared | Antibiotic resistance MIC regression subset, 56 antibiotics | R² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless | 3.95 | 19.7 | Mistral-DNA |
| BacBench essential gene prediction, linear probe AUPRC | BacBench essential gene dataset from the Database of Essential Genes | AUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless | 15.4 | 14.9 | Evo 2 |
| BacBench essential gene prediction, linear probe AUROC | BacBench essential gene dataset from the Database of Essential Genes | AUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless | 56 | 58.8 | Mistral-DNA |
| BacBench phenotypic trait prediction, linear probe AUPRC | BacBench phenotypic trait dataset | AUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless | 25 | 39.6 | Mistral-DNA |
| BacBench phenotypic trait prediction, linear probe AUROC | BacBench phenotypic trait dataset | AUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless | 55.4 | 60.3 | Mistral-DNA |
| BacBench protein-protein interaction, linear probe AUPRC | BacBench protein-protein interaction dataset from STRING DB | AUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless | 16 | 21.1 | Mistral-DNA |
| BacBench protein-protein interaction, linear probe AUROC | BacBench protein-protein interaction dataset from STRING DB | AUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless | 50 | 56 | Mistral-DNA |
Comparisons in GUANinE v1.1 reveals complementarity… (6)
| Protocol | Dataset | Metric | Evo 2 | Mistral-DNA | Higher |
|---|---|---|---|---|---|
| GUANinE v1.1 ccre-propensity, Spearman rho x100 (Table 2) | GUANinE ccre-propensity test set | Spearman (GUANinE v1.1 test set), unitless | 40.2 | 36.1 | Evo 2 |
| GUANinE v1.1 cons100, Spearman rho x100 (Table 2) | GUANinE cons100 test set | Spearman (GUANinE v1.1 test set), unitless | 55.2 | 50.7 | Evo 2 |
| GUANinE v1.1 cons30, Spearman rho x100 (Table 2) | GUANinE cons30 test set | Spearman (GUANinE v1.1 test set), unitless | 68.5 | 60.5 | Evo 2 |
| GUANinE v1.1 dnase-propensity, Spearman rho x100 (Table 2) | GUANinE dnase-propensity test set | Spearman (GUANinE v1.1 test set), unitless | 41.8 | 39.2 | Evo 2 |
| GUANinE v1.1 gpra-c, Spearman rho x100 (Table 2) | GUANinE gpra-c test set | Spearman (GUANinE v1.1 test set), unitless | 72.7 | 69.4 | Evo 2 |
| GUANinE v1.1 gpra-d, Spearman rho x100 (Table 2) | GUANinE gpra-d test set | Spearman (GUANinE v1.1 test set), unitless | 76.4 | 73.2 | Evo 2 |
What disagrees
- studies
- 2
- Compared
- Evo 2, Mistral-DNA
- Study that disagrees
- Other studies
- Rule
- Two-study conflict; different-task
- Model identity
- Matched across studies partly through links not yet reviewed
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-c971306227b3bb8e · All contested results · How findings are made and followed up