Relevance of vep-lit-20261014-protocol-proteingym-clinical-supervised-substitutions-auc-144fe22 to use-case-missense-pp3-bp4-calibrated-evidence

Stable ID: use-case-mapping-vep-uc-20261014-missense-pp3-bp4-calibrated-evidence-proteingym-clinical-supervised-substitutions-auc-144fe22

field

links:assessed_by:vep-lit-20261014-protocol-proteingym-clinical-supervised-substitutions-auc-144fe22

value

vep-lit-20261014-protocol-proteingym-clinical-supervised-substitutions-auc-144fe22

relevance

proxy

endpoint

Average over proteins of AUC for supervised clinical predictors from dbNSFP 4.4a

rationale

Proxy: includes most calibrated tools (for example REVEL, BayesDel, VEST4), but scores them on all ClinVar labels without thresholds.

constraints

Inspect every linked evaluation's source locator before citing a result.; Do not combine this mapping's evaluations with any other protocol's results.; Compare rows only within this protocol.

limitations

Do not compare with the zero-shot rows as if on equal footing.; Performance is reported on all available ClinVar labels, so supervised methods whose training data overlap these labels are likely overestimated; models trained on population frequencies leak further, because most ClinVar benign labels rest on observed frequency (ProteinGym paper, 'Clinical datasets', supervised benchmark).; Clinical labels are biased towards heavily studied proteins, such as well-known cancer genes, and towards European ancestry (ProteinGym paper, Limitations).; Evolutionary conservation is one of the criteria used to classify ClinVar variants, so even zero-shot models are not free of circularity (ProteinGym paper, Limitations).; Per-protein AUC averaged over proteins does not test whether scores are comparable across genes, which genome-wide thresholds assume.; The ProteinGym paper describes the 2023 clinical sets; whether the sets behind the 144fe22 files are unchanged is not stated.; Transcribed from the source; no reproduction by Rewire and no human scientific review.

citation locators

source id: vep-lit-20261014-source-proteingym-clinical-supervised-substitutions-auc-summary-144fe22; locator: Summary_performance_clinical_substitutions_AUC.csv (supervised); README line 106; source id: profile-protocol-proteingym-paper-xml-a58110e0; locator: ProteinGym NeurIPS 2023 paper (PMC10723403): 'Clinical datasets', Appendix A.3.2, Appendix 'Limitations' ('Bias', 'Circularity')

source locator

vep-lit-20261014-source-proteingym-clinical-supervised-substitutions-auc-summary-144fe22: Summary_performance_clinical_substitutions_AUC.csv (supervised); README line 106; profile-protocol-proteingym-paper-xml-a58110e0: ProteinGym NeurIPS 2023 paper (PMC10723403): 'Clinical datasets', Appendix A.3.2, Appendix 'Limitations' ('Bias', 'Circularity')

headline metric

auroc

comparison group

proteingym-clinical-144fe22

comparison title

ProteinGym clinical benchmark (commit 144fe22)

stratum label

Substitutions, supervised predictors

stratum order

2

reason

VEP use-case pass 2026-10-14: relevance of a protocol to a new use case (rewire-benchmark-data #151/#152).

review

reviewer: claude; reviewer note: Claude review agent; no human review claimed. Independent Claude (Opus 5.5) review agent, separate from the extraction agent that built this batch. Automated review only. Transcription, not reproduction.; reviewed at: 2026-10-14T16:00:00Z; method: ai-assisted-source-review; scope-review; method note: Checked the endpoint against the use case's question, inputs, exclusions and output; checked each limitation and number against the cited sources, re-downloaded and hash-matched; listed every evaluation on the protocol in reviewed_evaluations (values of evaluations already in the store were checked by their own independent reviews).; note: Judgement drafted by the extraction agent and revised in review (docs/reviews/vep-use-cases-2026-10-14-review.md).

reviewed evaluations

vep-lit-20261014-eval-pg-css-proteingym-clinical-bayesdel-addaf; vep-lit-20261014-eval-pg-css-proteingym-clinical-bayesdel-noaf; vep-lit-20261014-eval-pg-css-proteingym-clinical-cadd; vep-lit-20261014-eval-pg-css-proteingym-clinical-clinpred; vep-lit-20261014-eval-pg-css-proteingym-clinical-dann; vep-lit-20261014-eval-pg-css-proteingym-clinical-deogen2; vep-lit-20261014-eval-pg-css-proteingym-clinical-fathmm; vep-lit-20261014-eval-pg-css-proteingym-clinical-gmvp; vep-lit-20261014-eval-pg-css-proteingym-clinical-metarnn; vep-lit-20261014-eval-pg-css-proteingym-clinical-mpc; vep-lit-20261014-eval-pg-css-proteingym-clinical-mutationtaster; vep-lit-20261014-eval-pg-css-proteingym-clinical-polyphen2-hdiv; vep-lit-20261014-eval-pg-css-proteingym-clinical-polyphen2-hvar; vep-lit-20261014-eval-pg-css-proteingym-clinical-revel; vep-lit-20261014-eval-pg-css-proteingym-clinical-varity-er; vep-lit-20261014-eval-pg-css-proteingym-clinical-varity-r; vep-lit-20261014-eval-pg-css-proteingym-clinical-vest4

pins

record id: profile-protocol-proteingym-paper-xml-a58110e0; fields: status; attributes.artifact_sha256; sha256: 55a285515bd12e2145721382fccb111bba8ae70f3dd4e1c5aeae2afdc0c00e0d; record id: run-doc-proteingym-readme-md-144fe22b; fields: status; attributes.artifact_sha256; sha256: 70823ddfc9ef5ced58ba8aea340a8d99cf7808ee58b0d2cd4ee1abe5f6684646; record id: use-case-missense-pp3-bp4-calibrated-evidence; fields: status; name; description; facets; attributes.question; attributes.decision; attributes.inputs; attributes.output; attributes.exclusions; attributes.setting; attributes.clinical_scope; attributes.intended_users; sha256: 25250cac852da67b4c4b8ff30fab89588ca97898bcad5ccea87d30e669fea4d1; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-bayesdel-addaf; fields: status; links; attributes.origin; attributes.comparison; sha256: 6c924d3a7c112c6b4f2becb9be942c8d74765e223a607bd930b24f62240e6bbf; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-bayesdel-noaf; fields: status; links; attributes.origin; attributes.comparison; sha256: 7070b1c78a1ad77c29e6ebdbfe0f1c4153d4027d625830a78f22bbb2ae5c8729; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-cadd; fields: status; links; attributes.origin; attributes.comparison; sha256: 4da57aca7b85f7a9311d0b1e09ae27bd41bcfdf1a1898b850dfb3bd7c18fb557; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-clinpred; fields: status; links; attributes.origin; attributes.comparison; sha256: 262c32bb50554e6fe8f790b64a5edb927ed98e68269a1118119ac359e6937097; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-dann; fields: status; links; attributes.origin; attributes.comparison; sha256: 038c8e440d91d54b166a141ab94287e7d93593e19dd2f8192daaaa8f36173007; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-deogen2; fields: status; links; attributes.origin; attributes.comparison; sha256: 8f947e2f7e527250429177530207d68e0153a7bb5c154c3a7b755d64ef36ff53; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-fathmm; fields: status; links; attributes.origin; attributes.comparison; sha256: 0bee92945eb58a8445a90a4524031f87d05a281d688ac7d185a8682e2d777b6b; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-gmvp; fields: status; links; attributes.origin; attributes.comparison; sha256: df7e6911d56cfcadc952cceec4f19d39b54ff2200a3f3953ad9ba617b9173403; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-metarnn; fields: status; links; attributes.origin; attributes.comparison; sha256: 101cd6bce3310e280d756c297f39e498cad2a407b7b2435811d2b7ab95ab936e; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-mpc; fields: status; links; attributes.origin; attributes.comparison; sha256: 59ceffe04073817e19bfc7717d70e97b1b34cdf87dae0155b5f70b4f5af57430; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-mutationtaster; fields: status; links; attributes.origin; attributes.comparison; sha256: 76cfda7e2dc71c82c4f1d42a8e4a8c8cc24a55cfa3f1164c2443213e147b81ad; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-polyphen2-hdiv; fields: status; links; attributes.origin; attributes.comparison; sha256: 2ade03c922440ffccf415805d828009d4c1cff65fa0ab3775b056bf20f9a927b; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-polyphen2-hvar; fields: status; links; attributes.origin; attributes.comparison; sha256: 1e45f4a2621d2e2304fa9e478423c95d4a586eefe8ab6fc0af3f8de8c7a392d4; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-revel; fields: status; links; attributes.origin; attributes.comparison; sha256: 4e1ad2973e53cb7fc12092888661ff223eb3da7dd1ad1c99157060a6ef9a79f2; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-varity-er; fields: status; links; attributes.origin; attributes.comparison; sha256: 800e1677b31bae5e923088846179fd34a6a4f83b3d389513ce60fb5f6a2b19ce; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-varity-r; fields: status; links; attributes.origin; attributes.comparison; sha256: a65750a5c7a6d3c0adb664c6e90c89ac401dcfffa20da6a2f01c85a7d583ef08; record id: vep-lit-20261014-eval-pg-css-proteingym-clinical-vest4; fields: status; links; attributes.origin; attributes.comparison; sha256: dd3829f8ec1125f1319bf09d11fc212f5f29b96d5cc4da1ba8e46024d53b126b; record id: vep-lit-20261014-protocol-proteingym-clinical-supervised-substitutions-auc-144fe22; fields: status; name; description; facets; source_ids; links; attributes; sha256: 718c15cdbd4989f6033334b65319a6ef5dc8174489ca666e15c0d617b1a71d95; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-bayesdel-addaf-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 31f46b207e03fb624b8f041370a9cdef88be6444db746a61e5488efb54faa2d5; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-bayesdel-noaf-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 1e1d21ca91107afb01baa97bcbc72340fa570a9bdab0a3f66679427ce250ff42; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-cadd-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: cc86137192da5fa5e82e28e5ebe9cbf82a4c0a793f262ca4eb0d1f29688a421a; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-clinpred-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: c06d0b3e3e22b9b14f52da6e0939887f54828a74aca4636293fb8562cd3c75f2; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-dann-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: d3392a36dc37cf66cd550f062ce6714335ff05e6bfe05772d7949b7cbf92652b; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-deogen2-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: ea4f27cf29736926ded11a169ddfda181c0f917c7edf45f9971fc526df1b118b; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-fathmm-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: bef626c25a8a02edd8ee805e1d65275e0383ab824a4565a7ac240f983529f960; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-gmvp-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 8edac6c9fd0c985924411d4b48946c0decb12444f91097c794b857813adfd5a5; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-metarnn-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 9df7769c6e26d5b183945eb7d4663e5518d4a0e1919ff64f9099a7b9d571f7ef; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-mpc-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 9c40419816791656f6e8e33dfdce2a512809e8de68c661b14e7a8fae12aced57; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-mutationtaster-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 804d86de2fd3837588b63678a3b996697ce8812026beff519b98d4ee18301963; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-polyphen2-hdiv-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 00ce21bca775bbd3eda25c861404d9ee2ad32b7fc3a877445480c6668e1ab7e9; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-polyphen2-hvar-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 6a6926e19851c96e0528c18590fd0ecda297fc3840edc25836311ca422ada0e4; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-revel-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: d877a8c8f1da08fa842da13a6af62a615951bf5a3c8c7acf027429dce58ed62c; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-varity-er-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: efc539ec0fa1e3419132a2059998c972882e7145a84c2673ed1ff62e2b17e2b1; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-varity-r-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 7cd9eb94bc91d393b5d2ba6eb53cf7afab27f4c4eb278370a151a919a6efb40d; record id: vep-lit-20261014-result-pg-css-proteingym-clinical-vest4-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: eb11621dccd37c77511a22466033f3d173e64e65d4bf8f55ae071e975190d8ec; record id: vep-lit-20261014-source-proteingym-clinical-supervised-substitutions-auc-summary-144fe22; fields: status; attributes.artifact_sha256; sha256: a3b280aaeea6c608c1e79070e378c56f3f4480c88bf6ca5a568e93dea09c4c4b

• subject: Use calibrated computational evidence for missense variants (PP3/BP4)