ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22
144fe22b07dfaeec2b366f2346203a9838a55b4c · Project source
What this source contributed
Records in this release that cite this source. A record can cite several sources.
Results
Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.
Claims
Statements this source supports, each with where in the source it was checked.
- Relevance of vep-lit-20261014-protocol-proteingym-clinical-zero-shot-substitutions-auc-144fe22 to use-case-brca1-brca2-germline-interpretation: Assessed by ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22) (vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot))
- Relevance of vep-lit-20261014-protocol-proteingym-clinical-zero-shot-substitutions-auc-144fe22 to use-case-missense-pp3-bp4-calibrated-evidence: Assessed by ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22) (vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot); README lines 22-28, 106; profile-protocol-proteingym-paper-xml-a58110e0: ProteinGym NeurIPS 2023 paper (PMC10723403): 'Clinical datasets', Appendix A.3.2, Appendix 'Limitations' ('Bias', 'Circularity'))
- Relevance of vep-lit-20261014-protocol-proteingym-clinical-zero-shot-substitutions-auc-144fe22 to use-case-rare-disease-candidate-ranking: Assessed by ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22) (vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot); README lines 22-28, 106)
Models, benchmarks and other records
Configurations (27)
- BayesDel (addAF) [ProteinGym 144fe22 clinical summary]
- BayesDel (noAF) [ProteinGym 144fe22 clinical summary]
- CADD [ProteinGym 144fe22 clinical summary]
- ClinPred [ProteinGym 144fe22 clinical summary]
- DANN [ProteinGym 144fe22 clinical summary]
- DEOGEN2 [ProteinGym 144fe22 clinical summary]
- EVE [ProteinGym 144fe22 clinical summary]
- FATHMM [ProteinGym 144fe22 clinical summary]
- gMVP [ProteinGym 144fe22 clinical summary]
- Hidden Markov Model [ProteinGym 144fe22 clinical summary]
- LIST-S2 [ProteinGym 144fe22 clinical summary]
- LRT [ProteinGym 144fe22 clinical summary]
- MetaRNN [ProteinGym 144fe22 clinical summary]
- MPC [ProteinGym 144fe22 clinical summary]
- MutationAssessor [ProteinGym 144fe22 clinical summary]
- MutationTaster [ProteinGym 144fe22 clinical summary]
- MutPred [ProteinGym 144fe22 clinical summary]
- PolyPhen2 (HDIV) [ProteinGym 144fe22 clinical summary]
- PolyPhen2 (HVAR) [ProteinGym 144fe22 clinical summary]
- PrimateAI [ProteinGym 144fe22 clinical summary]
- Provean [ProteinGym 144fe22 clinical summary]
- REVEL [ProteinGym 144fe22 clinical summary]
- SIFT [ProteinGym 144fe22 clinical summary]
- SIFT4G [ProteinGym 144fe22 clinical summary]
- VARITY (ER) [ProteinGym 144fe22 clinical summary]
Protocols (1)
Evaluations (13)
- ESM-1b: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- EVE: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- GEMME: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- LIST-S2: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- LRT: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- MutationAssessor: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- MutPred: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- PoET (200M): ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- PrimateAI: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- Provean: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- SIFT: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- SIFT4G: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
- TranceptEVE L: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
26 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| attributes.artifact_sha256 fbe453d1aaa6c30a1d0adf610f7fc23c36044a8e2548b178d316777383f8fe93 Source metadata | ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 No field-specific location recorded Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Artifact url https://raw.githubusercontent.com/OATML-Markslab/ProteinGym/144fe22b07dfaeec2b366f2346203a9838a55b4c/benchmarks/clinical_zero_shot/substitutions/AUC/Summary_performance_clinical_substitutions_AUC.csv Source metadata | ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 No field-specific location recorded Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Licence MIT (repository LICENSE); underlying assay and clinical data carry their own terms Source metadata | ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 No field-specific location recorded Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Limitations4 values
| ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 No field-specific location recorded Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Media type text/csv Source metadata | ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 No field-specific location recorded Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Publication status official_project_source Source metadata | ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 No field-specific location recorded Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Retrieved at 2026-10-14T13:39:33Z Source metadata | ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 No field-specific location recorded Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| attributes.review.artifact_sha256 fbe453d1aaa6c30a1d0adf610f7fc23c36044a8e2548b178d316777383f8fe93 Source metadata | ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 No field-specific location recorded Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Review: method note Re-downloaded from the recorded URL on 2026-10-14 and matched the SHA-256; URL, version, licence and limitations checked. Source metadata | ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 No field-specific location recorded Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Review method2 values
| ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 No field-specific location recorded Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c | catalogued No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
Release 2026-10-22-1b7cf027e4f6 · Record review: source checked
Official ProteinGym summary file: average AUC on the clinical substitution benchmark for 13 zero-shot models and unsupervised clinical predictors.
Technical metadata and extraction receipts
Stable ID: vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22
- areas
- proteins-complexes
- contexts
- clinical_research
- url
- https://github.com/OATML-Markslab/ProteinGym/blob/144fe22b07dfaeec2b366f2346203a9838a55b4c/benchmarks/clinical_zero_shot/substitutions/AUC/Summary_performance_clinical_substitutions_AUC.csv
- artifact url
- https://raw.githubusercontent.com/OATML-Markslab/ProteinGym/144fe22b07dfaeec2b366f2346203a9838a55b4c/benchmarks/clinical_zero_shot/substitutions/AUC/Summary_performance_clinical_substitutions_AUC.csv
- version
- 144fe22b07dfaeec2b366f2346203a9838a55b4c
- retrieved at
- 2026-10-14T13:39:33Z
- artifact sha256
- fbe453d1aaa6c30a1d0adf610f7fc23c36044a8e2548b178d316777383f8fe93
- publication status
- official_project_source
- licence
- MIT (repository LICENSE); underlying assay and clinical data carry their own terms
- media type
- text/csv
- source locator
- benchmarks/clinical_zero_shot/substitutions/AUC/Summary_performance_clinical_substitutions_AUC.csv
- limitations
- Mutable repository: the file is pinned to commit 144fe22b07dfaeec2b366f2346203a9838a55b4c, which was the head of main at retrieval (committed 2026-03-25).; The repository does not record which script run, data release or model-score snapshot produced this file; the README says these summary files are what the website hosts.; The header repeats 'Model_rank' and the second 'Model_rank' column is empty on every row.; The file lists 13 models, far fewer than the zero-shot DMS file; the repository does not say whether the clinical summary is partial.
- review
- reviewer: claude; reviewer note: Claude review agent; no human review claimed. Independent Claude (Opus 5.5) review agent, separate from the extraction agent that built this batch. Automated review only. Transcription, not reproduction.; reviewed at: 2026-10-14T15:30:00Z; artifact sha256: fbe453d1aaa6c30a1d0adf610f7fc23c36044a8e2548b178d316777383f8fe93; retrieval url: https://raw.githubusercontent.com/OATML-Markslab/ProteinGym/144fe22b07dfaeec2b366f2346203a9838a55b4c/benchmarks/clinical_zero_shot/substitutions/AUC/Summary_performance_clinical_substitutions_AUC.csv; method: source-hash-verification; ai-assisted-source-review; method note: Re-downloaded from the recorded URL on 2026-10-14 and matched the SHA-256; URL, version, licence and limitations checked.; note: Hash matches.