rewirebio.iobenchmarks
Contested result

Evo 2 and Mistral-DNA: GUANinE v1.1 reveals complementarity of supervised and genomic language models and BacBench LICENSE file at commit 64b7d45 disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

GUANinE v1.1 reveals complementarity of supervised and genomic language models ranks Evo 2 above Mistral-DNA in 6 of 6 comparisons; BacBench LICENSE file at commit 64b7d45 ranks Mistral-DNA above Evo 2 in 9 of 10 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank Evo 2 above Mistral-DNA in 7 of their 16 comparisons and Mistral-DNA higher in 9. Of those papers, BacBench LICENSE file at commit 64b7d45 ranks Mistral-DNA higher on its own (9 of 10).

Papers grouped by which system they rank higher: Evo 2 or Mistral-DNARank Evo 2 higher: GUANinE v1.1 reveals complementarity…. Rank Mistral-DNA higher: BacBench LICENSE file at commit 64b7d45. Mixed: none.Rank Evo 2 higherMixedRank Mistral-DNA higherGUANinE v1.1 revealscomplementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language modelsBacBench LICENSE fileat commit 64b7d45BacBench LICENSE file at commit 64b7d45
  • Rank Evo 2 higher (1): GUANinE v1.1 reveals complementarity…
  • Rank Mistral-DNA higher (1): BacBench LICENSE file at commit 64b7d45
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Evo 2 or Mistral-DNA higherBacBench LICENSE file at commit 64b7d45: Evo 2 higher in 1, Mistral-DNA higher in 9, 0 ties. GUANinE v1.1 reveals complementarity…: Evo 2 higher in 6, Mistral-DNA higher in 0, 0 ties.BacBench LICENSE file at commit 64b7d45BacBench LICENSE file at commit 64b7d451 Evo 2 higher19 Mistral-DNA higher9GUANinE v1.1 reveals complementarity…GUANinE v1.1 reveals complementarity of supervised and genomic language models6 Evo 2 higher6
Evo 2 higherTieMistral-DNA higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks Evo 2 higherRanks Mistral-DNA higherTies
BacBench LICENSE file at commit 64b7d45 and 12 more records · DOIAgrees190
GUANinE v1.1 reveals complementarity of supervised and genomic language models · DOIAgrees600
Comparisons in BacBench LICENSE file at commit 64b7d45 (10)
ProtocolDatasetMetricEvo 2Mistral-DNAHigher
BacBench antibiotic resistance, binary linear probe AUPRCAntibiotic resistance binary prediction subset, 37 antibioticsAUPRC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless38.449.9Mistral-DNA
BacBench antibiotic resistance, binary linear probe AUROCAntibiotic resistance binary prediction subset, 37 antibioticsAUROC (linear head per drug on frozen genome embeddings, random 70/10/20 split, mean over 37 antibiotics), unitless66.376Mistral-DNA
BacBench antibiotic resistance, MIC regression Pearson correlationAntibiotic resistance MIC regression subset, 56 antibioticsPearson (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless0.5341Mistral-DNA
BacBench antibiotic resistance, MIC regression R squaredAntibiotic resistance MIC regression subset, 56 antibioticsR² (linear head per drug on frozen genome embeddings predicting log(1+MIC), random 70/10/20 split, mean over 56 antibiotics, printed as a percentage), unitless3.9519.7Mistral-DNA
BacBench essential gene prediction, linear probe AUPRCBacBench essential gene dataset from the Database of Essential GenesAUPRC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless15.414.9Evo 2
BacBench essential gene prediction, linear probe AUROCBacBench essential gene dataset from the Database of Essential GenesAUROC (linear head on frozen gene embeddings, genus-level split, median over held-out test genomes), unitless5658.8Mistral-DNA
BacBench phenotypic trait prediction, linear probe AUPRCBacBench phenotypic trait datasetAUPRC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless2539.6Mistral-DNA
BacBench phenotypic trait prediction, linear probe AUROCBacBench phenotypic trait datasetAUROC (linear head per trait on frozen genome embeddings, macro-averaged across classes of each trait, genus-level split, mean over traits and seeds), unitless55.460.3Mistral-DNA
BacBench protein-protein interaction, linear probe AUPRCBacBench protein-protein interaction dataset from STRING DBAUPRC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless1621.1Mistral-DNA
BacBench protein-protein interaction, linear probe AUROCBacBench protein-protein interaction dataset from STRING DBAUROC (linear head on frozen pair embeddings plus their cosine similarity, genus-level split, mean over held-out test genomes), unitless5056Mistral-DNA
Comparisons in GUANinE v1.1 reveals complementarity… (6)

What disagrees

studies
2
Compared
Evo 2, Mistral-DNA
Study that disagrees
Other studies
Rule
Two-study conflict; different-task
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-c971306227b3bb8e · All contested results · How findings are made and followed up