ProteinGym zero-shot DMS substitutions (commit 144fe22)
Direct evidenceSource does not say who ran it: The source prints the value but does not say who ran the tool, so it is not counted as independent.Average_Spearman: mean over proteins of assay-level Spearman correlation between zero-shot scores and DMS readouts, 217 substitution assays, with breakdowns by MSA depth, function type, taxon and mutational depth
Spearman (higher is better), with the other metrics below each value. Showing the top 10 of 44 rows. The best value in each column is highlighted.
| Tool | All assays (Average_Spearman) and breakdowns |
|---|---|
AIDO Protein-RAG (16B)foundation model | 0.518 (best in column)mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
VenusREMfoundation model | 0.518 (best in column)mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
| 0.507mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
S3F-MSAfoundation model | 0.496mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
S2F-MSAfoundation model | 0.488mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
Protrieverfoundation model | 0.479mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
| 0.476mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
PoET (200M)foundation model | 0.470mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
S3Ffoundation model | 0.470mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
| 0.466mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
| 0.465mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
VespaGmachine learning | 0.458mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
| 0.457mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
S2Ffoundation model | 0.456mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
TranceptEVE L3 sweeps | 0.456mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
| 0.455mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
| 0.449mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
EVE (ensemble)2 sweeps | 0.439mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
VESPAmachine learning | 0.435mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
| 0.434mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
| 0.432mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
| 0.424mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
ESM-IF1foundation model | 0.422mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
DeepSequence (ensemble)2 sweeps | 0.419mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
| 0.414mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
| 0.407mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
| 0.406mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
MIF-STfoundation model | 0.400mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
| 0.399mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
| 0.395mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
ESM-1bfoundation model | 0.394mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
VESPAlmachine learning | 0.393mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
| 0.392mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
| 0.391mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
MIFfoundation model | 0.383mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
| 0.373mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
| 0.373mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
| 0.369mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) | |
Site-IndependentBaseline | 0.359mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
Unirep evotunedfoundation model | 0.347mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
MULANfoundation model | 0.317mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
ProteinMPNNfoundation model | 0.257mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
Unirepfoundation model | 0.190mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
ProtGPT2foundation model | 0.188mean over selection types of the per-type mean over proteins of assay-level Spearman (all 217 assays) |
Show all 44 rowsShow the top 10 rows only
Caveats and method
- Averages over proteins do not predict performance on a single protein of interest.
- Leaderboard rows were computed by the ProteinGym maintainers with their scoring scripts (origin unreported); no Rewire run on the full track.
- Assays include multiple substitutions; Depth_1 is the singles-only column.
- Bootstrap standard errors are relative to the top model (the top row prints 0.0).
- Some leaderboard models were developed by ProteinGym's authors (for example EVE, Tranception and TranceptEVE).
- Assays come from proteins of many organisms; results for human proteins alone are not given in this summary.
- Transcribed from the source; no reproduction by Rewire and no human scientific review.