Discrepancy investigations
Examine why benchmark results differ, test possible explanations, and identify which findings need independent evidence.
Reviewed reports
Release 2026-11-02-a7f03c9e55a5 · 0 reviewed reports
No reviewed investigations are available in this release. Completed and staged runs become public only after their evidence and interpretation have been reviewed.
Disagreements between studies
Each release flags studies that rank models against the order other studies agree on. These automatic leads, and what we are doing about each, are listed under contested results.
We have opened an investigation for these:
- ESM-2 and ESM3: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and pfmbench primary benchmark evidence disagree · Under investigation · Investigation
- DNABERT-2 and RNA-FM: mRNABench: A curated benchmark for mature mRNA property and function prediction reverses the order · Explained · Investigation
- ESM-1b and EVE: Brandes et al. 2023 Supplementary Table 2: full benchmark evaluations of ESM1b and all other methods reverses the order · Under investigation · Investigation
- RNAErnie and AIDO.RNA: Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning and RNAGym fitness leaderboard (v0.1.1), leaderboard/fitness/README.md disagree · Explained · Investigation
- ESM-IF1 and ProteinMPNN: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and proteinbench primary benchmark evidence disagree · Explained · Investigation
- ESM3 and ProteinMPNN: ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design and proteinbench primary benchmark evidence disagree · Explained · Investigation
- Geneformer and scGPT: resolved · Explained · Investigation
- AlphaGenome and Borzoi: resolved · Confirmed error · Investigation
- ESM-1v and ESM-2: resolved · Not a real disagreement · Investigation
- Geneformer and scGPT: resolved · Explained · Investigation
- Mutect2 and Octopus: resolved · Explained · Investigation
- Resolved discrepancy · Not a real disagreement · Investigation
- ESM3 and ProteinMPNN: resolved · Explained · Investigation
What an investigation establishes
A discrepancy can arise from a data error, metric definition, sample population, model limitation or biological context. We check the data and evaluation procedure before interpreting an unexpected score as a biological effect.
Each report connects its question to exact artifacts, a frozen analysis plan, every attempted explanation and its remaining uncertainty. All current reports are exploratory. Independent biological validation requires a separate study with a hypothesis fixed before examining the validation data.
Start with the evidence
Dataset and evaluation pages assess four separate capabilities: replay metrics, investigate discrepancies, run locally and validate independently. Missing evidence is listed explicitly. Complete evidence does not imply that the necessary files are installed on your computer.
Explore datasets and evidence gaps
Download readiness assessments (gzip) · Artifact manifests (gzip) · Reviewed reports (gzip)