ESM-2 and ESM3: pfmbench primary benchmark evidence reverses the order

Which studies agree

ProteinGym AMFR assay-level Spearman… ranks ESM3 above ESM-2 in 17 of its 19 comparisons and ESM-2 higher in 1; 1 is a tie. Pfmbench primary benchmark evidence ranks ESM-2 above ESM3 in 9 of its 12 comparisons and ESM3 higher in 3.

What disagrees

Other studies

rank ESM3 above ESM-2 in 17 of 18 comparisons

This study

ranks ESM-2 above ESM3 in 9 of 12 comparisons

Compared

ESM-2, ESM3

Study that disagrees

pfmbench primary benchmark evidence

Other studies

ProteinGym AMFR assay-level Spearman comparison, ProteinGym clinical supervised substitutions AUC summary at commit 144fe22, ProteinGym clinical zero-shot indels AUC summary at commit 144fe22, ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22, ProteinGym zero-shot DMS substitutions Spearman summary at commit 144fe22

Rule

Ordering reversal; different tasks across studies

Model identity

Matched across studies through reviewed links

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-4db3e5d24233e19b · All contested results · How findings are made and followed up