rewirebio.iobenchmarks
Evidence and investigation

Contested results

Where one study disagrees with what other studies report for the same models, and what we are doing to resolve each case.

These are automatic leads, not findings of error. Each release, a check compares how every study ranks the models it shares with other studies. It flags a study that reverses the order the others agree on, or whose gap to a shared reference model lies well outside theirs. Different tasks, populations or metrics often explain a flag.

A flag never hides a result. Flagged results stay in every table, page and download, marked Contested. A result is withheld only when a reviewer marks it disputed.

How the check works, and how each follow-up is recorded: discrepancies between studies.

In this release

  • 220open
  • 0settled
  • 1resolved
  • 1high severity
  • 12medium severity
  • 208low severity

Release 2026-10-23-5347695500ed · 221 findings in all · rules version 3

Findings

Status: All · Open · Settled · Resolved

Severity: All · High · Medium · Low

State: All · Not yet reviewed · Confirmed error

1 finding matches, most severe first.

ESM-IF1 and ProteinMPNN: proteinbench primary benchmark evidence reverses the order

  • High severity
  • Not yet reviewed
  • Ordering reversal
  • Different tasks across studies

proteinbench primary benchmark evidence ranks ProteinMPNN above ESM-IF1 in 10 of 10 comparisons; other studies rank ESM-IF1 above ProteinMPNN in 24 of 24.

2 papers rank ESM-IF1 above ProteinMPNN in all 24 of their comparisons. Proteinbench primary benchmark evidence ranks ProteinMPNN above ESM-IF1 in all 10 of its comparisons.

Comparisons in which each paper scores ESM-IF1 or ProteinMPNN higher2 other papers: ESM-IF1 higher in 24, ProteinMPNN higher in 0, 0 ties. proteinbench primary benchmark evidence (disagrees): ESM-IF1 higher in 0, ProteinMPNN higher in 10, 0 ties.2 other papers24 ESM-IF1 higher24Disagrees: proteinbench primary benchmark evidenceproteinbench primary benchmark evidence10 ProteinMPNN higher10
ESM-IF1 higherTieProteinMPNN higher
Compared
ESM-IF1, ProteinMPNN

Not reviewed yet.