ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 38 configurations
• 1 benchmark
• 18 tasks
• 2 dataset subsets
• 221 evaluations
• 221 results
• 106 evidence claims

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• CARP (640M) on ProteinGym ZS-SUB-AUC: Zero-shot substitutions, AUC: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• CARP (640M) on ProteinGym ZS-SUB-MCC: Zero-shot substitutions, MCC: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• CARP (640M) on ProteinGym ZS-SUB-NDCG: Zero-shot substitutions, NDCG@10%: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• CARP (640M) on ProteinGym ZS-SUB-RECALL: Zero-shot substitutions, top 10% recall: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• CARP (640M) on ProteinGym ZS-SUB-SPEARMAN: Zero-shot substitutions, Spearman: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• Embed. ESM-1v: uses model ESM-1v: Uses model ESM-1v (Section 4.2 Supervised benchmarks, Baselines; Table 3)
• ESM-1b on ProteinGym ZS-SUB-AUC: Zero-shot substitutions, AUC: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-1b on ProteinGym ZS-SUB-MCC: Zero-shot substitutions, MCC: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-1b on ProteinGym ZS-SUB-NDCG: Zero-shot substitutions, NDCG@10%: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-1b on ProteinGym ZS-SUB-RECALL: Zero-shot substitutions, top 10% recall: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-1b on ProteinGym ZS-SUB-SPEARMAN: Zero-shot substitutions, Spearman: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-1v (ensemble) on ProteinGym ZS-SUB-AUC: Zero-shot substitutions, AUC: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-1v (ensemble) on ProteinGym ZS-SUB-MCC: Zero-shot substitutions, MCC: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-1v (ensemble) on ProteinGym ZS-SUB-NDCG: Zero-shot substitutions, NDCG@10%: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-1v (ensemble) on ProteinGym ZS-SUB-RECALL: Zero-shot substitutions, top 10% recall: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-1v (ensemble) on ProteinGym ZS-SUB-SPEARMAN: Zero-shot substitutions, Spearman: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-1v (ensemble): family ESM-1v: Family ESM-1v (Section 4.1 Zero-shot benchmarks; Table 2 named model row)
• ESM-2 (15B) on ProteinGym ZS-SUB-AUC: Zero-shot substitutions, AUC: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-2 (15B) on ProteinGym ZS-SUB-MCC: Zero-shot substitutions, MCC: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-2 (15B) on ProteinGym ZS-SUB-NDCG: Zero-shot substitutions, NDCG@10%: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-2 (15B) on ProteinGym ZS-SUB-RECALL: Zero-shot substitutions, top 10% recall: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-2 (15B) on ProteinGym ZS-SUB-SPEARMAN: Zero-shot substitutions, Spearman: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)
• ESM-2 (15B): family ESM-2: Family ESM-2 (Section 4.1 Zero-shot benchmarks; Table 2 named model row)
• ESM-2: family ESM-2: Family ESM-2 (Section 4.1 zero-shot baselines; Table 2 ESM-2 (15B) and ProteinMPNN rows)
• ESM-IF1 on ProteinGym ZS-SUB-AUC: Zero-shot substitutions, AUC: attributes.origin: Origin independent_paper (Section 4.1 'Baselines': 'We implement a diverse set of 50+ zero-shot baselines'; Section 4.2 'Baselines' and Appendix A.4.2: supervised baselines 'defined in Notin et al. [2023]'; Appendix A.3.3: 'all of the model predictions can be reproduced using this repository'; Appendix A.4.1 per-baseline scoring; reference list)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.