Evidence claim

Relevance of vep-lit-20261014-protocol-proteingym-clinical-zero-shot-substitutions-auc-144fe22 to use-case-brca1-brca2-germline-interpretation

Use case: Interpret BRCA1/BRCA2 germline variants
Assessed by: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22)
Relevance: Proxy evidence
Endpoint: Average over proteins of AUC, pathogenic versus benign clinical substitutions

Relevance judgement drafted by the extraction agent; independently reviewed, automated review only.

13 reviewed evaluations

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

57 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-22-1b7cf027e4f6
Property and statement	Original source and location	Review and provenance
Citation locations 2 values source id vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22 locator Summary_performance_clinical_substitutions_AUC.csv (zero-shot) source id profile-protocol-proteingym-paper-xml-a58110e0 locator ProteinGym NeurIPS 2023 paper (PMC10723403): 'Clinical datasets' (zero-shot and supervised benchmarks), Appendix A.3.2 'Clinical datasets', Appendix 'Limitations' ('Bias', 'Circularity') Context-only references	Protocol metadata evidence: proteingym-paper.xml Original source ↗ vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC10723403 retrieved 2026-09-23 Retrieved: 2026-09-23T18:43:10.260678+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.citation_locators` Source artifact SHA-256: `a58110e02c3efc8395c534d48630f577c3d45e90d48198b7f9989cf52600b045` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Citation locations 2 values source id vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22 locator Summary_performance_clinical_substitutions_AUC.csv (zero-shot) source id profile-protocol-proteingym-paper-xml-a58110e0 locator ProteinGym NeurIPS 2023 paper (PMC10723403): 'Clinical datasets' (zero-shot and supervised benchmarks), Appendix A.3.2 'Clinical datasets', Appendix 'Limitations' ('Bias', 'Circularity') Context-only references	OATML-Markslab/ProteinGym / README.md Original source ↗ vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-17T09:32:03.517591+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.citation_locators` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Citation locations 2 values source id vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22 locator Summary_performance_clinical_substitutions_AUC.csv (zero-shot) source id profile-protocol-proteingym-paper-xml-a58110e0 locator ProteinGym NeurIPS 2023 paper (PMC10723403): 'Clinical datasets' (zero-shot and supervised benchmarks), Appendix A.3.2 'Clinical datasets', Appendix 'Limitations' ('Bias', 'Circularity') Context-only references	ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 Original source ↗ vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-10-14T13:39:33Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.citation_locators` Source artifact SHA-256: `fbe453d1aaa6c30a1d0adf610f7fc23c36044a8e2548b178d316777383f8fe93` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Comparison group proteingym-clinical-144fe22 Context-only references	Protocol metadata evidence: proteingym-paper.xml Original source ↗ vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC10723403 retrieved 2026-09-23 Retrieved: 2026-09-23T18:43:10.260678+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.comparison_group` Source artifact SHA-256: `a58110e02c3efc8395c534d48630f577c3d45e90d48198b7f9989cf52600b045` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Comparison group proteingym-clinical-144fe22 Context-only references	OATML-Markslab/ProteinGym / README.md Original source ↗ vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-17T09:32:03.517591+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.comparison_group` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Comparison group proteingym-clinical-144fe22 Context-only references	ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 Original source ↗ vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-10-14T13:39:33Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.comparison_group` Source artifact SHA-256: `fbe453d1aaa6c30a1d0adf610f7fc23c36044a8e2548b178d316777383f8fe93` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Comparison title ProteinGym clinical benchmark (commit 144fe22) Context-only references	Protocol metadata evidence: proteingym-paper.xml Original source ↗ vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC10723403 retrieved 2026-09-23 Retrieved: 2026-09-23T18:43:10.260678+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.comparison_title` Source artifact SHA-256: `a58110e02c3efc8395c534d48630f577c3d45e90d48198b7f9989cf52600b045` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Comparison title ProteinGym clinical benchmark (commit 144fe22) Context-only references	OATML-Markslab/ProteinGym / README.md Original source ↗ vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: Git commit 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-09-17T09:32:03.517591+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.comparison_title` Source artifact SHA-256: `321487a8de52c6cfa647a658f61150dd72e0acb0125470524fb94d1f8b23321a` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Comparison title ProteinGym clinical benchmark (commit 144fe22) Context-only references	ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 Original source ↗ vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 144fe22b07dfaeec2b366f2346203a9838a55b4c Retrieved: 2026-10-14T13:39:33Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.comparison_title` Source artifact SHA-256: `fbe453d1aaa6c30a1d0adf610f7fc23c36044a8e2548b178d316777383f8fe93` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Constraints 3 values Inspect every linked evaluation's source locator before citing a result. Do not combine this mapping's evaluations with any other protocol's results. Compare rows only within this protocol. Context-only references	Protocol metadata evidence: proteingym-paper.xml Original source ↗ vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot) Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: PMC10723403 retrieved 2026-09-23 Retrieved: 2026-09-23T18:43:10.260678+00:00	not individually reviewed No individual claim review recorded Audit details Field: `attributes.constraints` Source artifact SHA-256: `a58110e02c3efc8395c534d48630f577c3d45e90d48198b7f9989cf52600b045` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact

How to read the evidence

Sources and history

Release 2026-10-22-1b7cf027e4f6 · Record review: source checked

3 source records and release history

ProteinGym clinical zero-shot substitutions AUC summary at commit 144fe22 · Original source · 144fe22b07dfaeec2b366f2346203a9838a55b4c
OATML-Markslab/ProteinGym / README.md · Original source · Git commit 144fe22b07dfaeec2b366f2346203a9838a55b4c
Protocol metadata evidence: proteingym-paper.xml · Original source · PMC10723403 retrieved 2026-09-23

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: use-case-mapping-vep-lit-20261014-brca1-brca2-germline-interpretation-proteingym-clinical-zero-shot-substitutions-auc-144fe22

field: links:assessed_by:vep-lit-20261014-protocol-proteingym-clinical-zero-shot-substitutions-auc-144fe22
value: vep-lit-20261014-protocol-proteingym-clinical-zero-shot-substitutions-auc-144fe22
relevance: proxy
endpoint: Average over proteins of AUC, pathogenic versus benign clinical substitutions
rationale: Proxy: pan-gene discrimination of clinical missense variants; not BRCA-specific and not calibrated to evidence strength.
constraints: Inspect every linked evaluation's source locator before citing a result.; Do not combine this mapping's evaluations with any other protocol's results.; Compare rows only within this protocol.
limitations: Only 13 models are listed in this file at 144fe22; the file's completeness is not stated.; Clinical labels are biased towards heavily studied proteins, such as well-known cancer genes, and towards European ancestry (ProteinGym paper, Limitations).; Evolutionary conservation is one of the criteria used to classify ClinVar variants, so even zero-shot models are not free of circularity (ProteinGym paper, Limitations).; Per-protein AUC averaged over proteins does not test whether scores are comparable across genes.; The ProteinGym paper describes the 2023 clinical sets; whether the sets behind the 144fe22 files are unchanged is not stated.; Whether BRCA1 or BRCA2 is among the clinical proteins is not stated in the summary files.; Transcribed from the source; no reproduction by Rewire and no human scientific review.
citation locators: source id: vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22; locator: Summary_performance_clinical_substitutions_AUC.csv (zero-shot); source id: profile-protocol-proteingym-paper-xml-a58110e0; locator: ProteinGym NeurIPS 2023 paper (PMC10723403): 'Clinical datasets' (zero-shot and supervised benchmarks), Appendix A.3.2 'Clinical datasets', Appendix 'Limitations' ('Bias', 'Circularity')
source locator: vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (zero-shot)
headline metric: auroc
comparison group: proteingym-clinical-144fe22
comparison title: ProteinGym clinical benchmark (commit 144fe22)
stratum label: Substitutions, zero-shot
stratum order: 1
reason: VEP literature pass 2026-10-14: relevance of a newly extracted protocol (rewire-benchmark-data issues #153-#155).
review: reviewer: claude; reviewer note: Claude review agent; no human review claimed. Independent Claude (Opus 5.5) review agent, separate from the extraction agent that built this batch. Automated review only. Transcription, not reproduction.; reviewed at: 2026-10-14T15:30:00Z; method: ai-assisted-source-review; scope-review; method note: Checked the endpoint against the use case's question, inputs, exclusions and setting in data/entities/use-cases.jsonl; checked each limitation against the cited sources; listed every evaluation on the protocol in reviewed_evaluations after review/check_values.py matched its results.; note: Judgement drafted by the extraction agent and revised in review (docs/reviews/vep-literature-2026-10-14-review.md). Proxy is justified; see the dated review.
reviewed evaluations: vep-lit-20261014-eval-pg-czs-esm-1b; vep-lit-20261014-eval-pg-czs-gemme; vep-lit-20261014-eval-pg-czs-poet-200m; vep-lit-20261014-eval-pg-czs-proteingym-clinical-eve; vep-lit-20261014-eval-pg-czs-proteingym-clinical-list-s2; vep-lit-20261014-eval-pg-czs-proteingym-clinical-lrt; vep-lit-20261014-eval-pg-czs-proteingym-clinical-mutationassessor; vep-lit-20261014-eval-pg-czs-proteingym-clinical-mutpred; vep-lit-20261014-eval-pg-czs-proteingym-clinical-primateai; vep-lit-20261014-eval-pg-czs-proteingym-clinical-provean; vep-lit-20261014-eval-pg-czs-proteingym-clinical-sift; vep-lit-20261014-eval-pg-czs-proteingym-clinical-sift4g; vep-lit-20261014-eval-pg-czs-trancepteve-l
pins: record id: profile-protocol-proteingym-paper-xml-a58110e0; fields: status; attributes.artifact_sha256; sha256: 55a285515bd12e2145721382fccb111bba8ae70f3dd4e1c5aeae2afdc0c00e0d; record id: run-doc-proteingym-readme-md-144fe22b; fields: status; attributes.artifact_sha256; sha256: 70823ddfc9ef5ced58ba8aea340a8d99cf7808ee58b0d2cd4ee1abe5f6684646; record id: use-case-brca1-brca2-germline-interpretation; fields: status; name; description; facets; attributes.question; attributes.decision; attributes.inputs; attributes.output; attributes.exclusions; attributes.setting; attributes.clinical_scope; attributes.intended_users; sha256: 9681ae92e727dde7a801aec8d4884ec28b009d134a52d806a266f7d3b64bd5bd; record id: vep-lit-20261014-eval-pg-czs-esm-1b; fields: status; links; attributes.origin; attributes.comparison; sha256: 2202c431ebfc1e5a4ff3e833b96036d5c4fdbb212c74a44b936607b35f26293c; record id: vep-lit-20261014-eval-pg-czs-gemme; fields: status; links; attributes.origin; attributes.comparison; sha256: 839c5f841b5244a6d471085546c7781ae7335b28296cc658bf1fea656958dd55; record id: vep-lit-20261014-eval-pg-czs-poet-200m; fields: status; links; attributes.origin; attributes.comparison; sha256: d5f991b6fa85ffedef4e7dd7561756bc343cdf23986ebd9405e1ceb056798072; record id: vep-lit-20261014-eval-pg-czs-proteingym-clinical-eve; fields: status; links; attributes.origin; attributes.comparison; sha256: d425c32df3edc8f6228167dd7eea309c3df94b453b30331909d0239642e75b41; record id: vep-lit-20261014-eval-pg-czs-proteingym-clinical-list-s2; fields: status; links; attributes.origin; attributes.comparison; sha256: b54efb11eb2e9299f183a6127c9c1620bbc76017d2e8a6fbb850494070d0f931; record id: vep-lit-20261014-eval-pg-czs-proteingym-clinical-lrt; fields: status; links; attributes.origin; attributes.comparison; sha256: cdf118fd11b0580e748635594cb5196d78347b0e1df169d075c2acc57d02b702; record id: vep-lit-20261014-eval-pg-czs-proteingym-clinical-mutationassessor; fields: status; links; attributes.origin; attributes.comparison; sha256: 218a2d3debb5dd83637a24c72b90083a3dfdf2fd2dd128281a06720d9f0245db; record id: vep-lit-20261014-eval-pg-czs-proteingym-clinical-mutpred; fields: status; links; attributes.origin; attributes.comparison; sha256: 0b0859c4acf072688eba2f261e8ce0e364a5bfde8550c996ab58d8905cc96ca2; record id: vep-lit-20261014-eval-pg-czs-proteingym-clinical-primateai; fields: status; links; attributes.origin; attributes.comparison; sha256: 032da5d47df493637bd4a5832031dc600a48fa8efda6b4697cf1b28e05f15b97; record id: vep-lit-20261014-eval-pg-czs-proteingym-clinical-provean; fields: status; links; attributes.origin; attributes.comparison; sha256: 61e3ba8446a5b7ea4317f90d20312a52778bc9eba3339bd4718aef5736114b83; record id: vep-lit-20261014-eval-pg-czs-proteingym-clinical-sift; fields: status; links; attributes.origin; attributes.comparison; sha256: 13842b9957d3248d9cb8f05222609793ef0804cd72b22aa254f6370f1072b773; record id: vep-lit-20261014-eval-pg-czs-proteingym-clinical-sift4g; fields: status; links; attributes.origin; attributes.comparison; sha256: 4c1c7632e7f99cc6d029389f411707f7164ca0d9c39cfec368a92e79f011b953; record id: vep-lit-20261014-eval-pg-czs-trancepteve-l; fields: status; links; attributes.origin; attributes.comparison; sha256: ed96c421bce8e5c238051d53ea04277559ccdb94d392b9d282714c4e9c6f8f21; record id: vep-lit-20261014-protocol-proteingym-clinical-zero-shot-substitutions-auc-144fe22; fields: status; name; description; facets; source_ids; links; attributes; sha256: ad9d453b5e4517d88969e8fc884f0ac01dee88793dd5a34c104aa58082cda754; record id: vep-lit-20261014-result-pg-czs-esm-1b-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 2039b0909a7a3c6a0b45d59dad74415dd84f04a1c23ea86b98561cc369e393ac; record id: vep-lit-20261014-result-pg-czs-gemme-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: b5ff018449801593aa7a650491a94d23f761e8c51cf4c974c046cc1f1e5d5b5d; record id: vep-lit-20261014-result-pg-czs-poet-200m-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: b5dd5d560c15cf79ca1439f7b7c31f3c6f4506959c07153041e77e737795bd15; record id: vep-lit-20261014-result-pg-czs-proteingym-clinical-eve-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 3dcdd7baa1ea87f043822bd40e6e3e2bdfc8425ddc37ed9cf609429a679ce47e; record id: vep-lit-20261014-result-pg-czs-proteingym-clinical-list-s2-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 001f8f89d47a794aa77674be8f6dd981135eeb57e0d29cdcef7f4b3b1fa6200a; record id: vep-lit-20261014-result-pg-czs-proteingym-clinical-lrt-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 88b9eaaca9fb2e8164e2f215761a1ccb97be25b40d93220cf0900b56ebd84486; record id: vep-lit-20261014-result-pg-czs-proteingym-clinical-mutationassessor-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: e4f10443cccc63a32b8d5829c7bc21d1d6c7e8e707877dec57ef68bc34d4ffcb; record id: vep-lit-20261014-result-pg-czs-proteingym-clinical-mutpred-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 617e1802d255672480db1a5ccc547b95cbf69e16df9659c8b43adfe271099dc8; record id: vep-lit-20261014-result-pg-czs-proteingym-clinical-primateai-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: df9b394a42f2fb578a3fd846818a35f268b29f08768098d4a9a40df6ba25d2cc; record id: vep-lit-20261014-result-pg-czs-proteingym-clinical-provean-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: aa80e3e48bd07eca2a0461aa1c7830c0a685ea57a7f89c5c85dabf74992378e0; record id: vep-lit-20261014-result-pg-czs-proteingym-clinical-sift-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 994ff53f99311a05d16f8bf42ad277e034627019601539d80af8b85d9f1cb335; record id: vep-lit-20261014-result-pg-czs-proteingym-clinical-sift4g-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: e4f10443cccc63a32b8d5829c7bc21d1d6c7e8e707877dec57ef68bc34d4ffcb; record id: vep-lit-20261014-result-pg-czs-trancepteve-l-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: b5dd5d560c15cf79ca1439f7b7c31f3c6f4506959c07153041e77e737795bd15; record id: vep-lit-20261014-source-proteingym-clinical-zero-shot-substitutions-auc-summary-144fe22; fields: status; attributes.artifact_sha256; sha256: 9c63637802906d90b2632a172c3074d9d3aa85ff367c59271ff195ff05a61293

Related records

subject: Interpret BRCA1/BRCA2 germline variants

Suggest a correction