Relevance of vep-lit-20261014-protocol-proteingym-clinical-zero-shot-indels-auc-144fe22 to use-case-rare-disease-candidate-ranking

Stable ID: use-case-mapping-vep-lit-20261014-rare-disease-candidate-ranking-proteingym-clinical-zero-shot-indels-auc-144fe22

field

links:assessed_by:vep-lit-20261014-protocol-proteingym-clinical-zero-shot-indels-auc-144fe22

value

vep-lit-20261014-protocol-proteingym-clinical-zero-shot-indels-auc-144fe22

relevance

proxy

endpoint

Average over proteins of AUC separating pathogenic from benign clinical indels

rationale

Proxy: discrimination of pathogenic short in-frame indels bears on prioritising indel candidates, which most missense predictors cannot score. It does not measure causal-variant recovery in cases.

constraints

Inspect every linked evaluation's source locator before citing a result.; Do not combine this mapping's evaluations with any other protocol's results.; Compare rows only within this protocol.

limitations

No uncertainty column in this file.; Benign indels are common gnomAD indels used as pseudocontrols, because 84% of ClinVar indel annotations are pathogenic; the task partly separates pathogenic from common variants (ProteinGym paper, 'Clinical datasets' and Appendix A.3.2).; Only short in-frame indels (up to 3 amino acids) are included; frameshifts were filtered out (Appendix A.3.2).; Clinical labels are biased towards heavily studied proteins, such as well-known cancer genes, and towards European ancestry (ProteinGym paper, Limitations).; Evolutionary conservation is one of the criteria used to classify ClinVar variants, so even zero-shot models are not free of circularity (ProteinGym paper, Limitations).; Per-protein AUC averaged over proteins does not test whether scores are comparable across genes.; The ProteinGym paper describes the 2023 clinical sets; whether the sets behind the 144fe22 files are unchanged is not stated.; The use case excludes balanced pathogenic/benign classification as a measure of case-level diagnostic performance; this protocol is variant-level classification and does not measure recovery of causal variants in cases or review effort.; Transcribed from the source; no reproduction by Rewire and no human scientific review.

citation locators

source id: vep-lit-20261014-source-proteingym-clinical-zero-shot-indels-auc-summary-144fe22; locator: Summary_performance_clinical_indels_AUC.csv; README line 22; source id: profile-protocol-proteingym-paper-xml-a58110e0; locator: ProteinGym NeurIPS 2023 paper (PMC10723403): 'Clinical datasets' (zero-shot and supervised benchmarks), Appendix A.3.2 'Clinical datasets', Appendix 'Limitations' ('Bias', 'Circularity')

source locator

vep-lit-20261014-source-proteingym-clinical-zero-shot-indels-auc-summary-144fe22: Summary_performance_clinical_indels_AUC.csv; README line 22

headline metric

auroc

comparison group

proteingym-clinical-144fe22

comparison title

ProteinGym clinical benchmark (commit 144fe22)

stratum label

Indels, zero-shot

stratum order

3

reason

VEP literature pass 2026-10-14: relevance of a newly extracted protocol (rewire-benchmark-data issues #153-#155).

review

reviewer: claude; reviewer note: Claude review agent; no human review claimed. Independent Claude (Opus 5.5) review agent, separate from the extraction agent that built this batch. Automated review only. Transcription, not reproduction.; reviewed at: 2026-10-14T15:30:00Z; method: ai-assisted-source-review; scope-review; method note: Checked the endpoint against the use case's question, inputs, exclusions and setting in data/entities/use-cases.jsonl; checked each limitation against the cited sources; listed every evaluation on the protocol in reviewed_evaluations after review/check_values.py matched its results.; note: Judgement drafted by the extraction agent and revised in review (docs/reviews/vep-literature-2026-10-14-review.md). Proxy is justified; see the dated review.

reviewed evaluations

vep-lit-20261014-eval-pg-czi-poet-200m; vep-lit-20261014-eval-pg-czi-progen2-base; vep-lit-20261014-eval-pg-czi-progen2-l; vep-lit-20261014-eval-pg-czi-progen2-m; vep-lit-20261014-eval-pg-czi-progen2-s; vep-lit-20261014-eval-pg-czi-progen2-xl; vep-lit-20261014-eval-pg-czi-proteingym-clinical-hidden-markov-model; vep-lit-20261014-eval-pg-czi-proteingym-clinical-provean; vep-lit-20261014-eval-pg-czi-protgpt2; vep-lit-20261014-eval-pg-czi-rita-l; vep-lit-20261014-eval-pg-czi-rita-m; vep-lit-20261014-eval-pg-czi-rita-s; vep-lit-20261014-eval-pg-czi-rita-xl; vep-lit-20261014-eval-pg-czi-tranception-l; vep-lit-20261014-eval-pg-czi-tranception-l-no-retrieval; vep-lit-20261014-eval-pg-czi-tranception-m; vep-lit-20261014-eval-pg-czi-tranception-m-no-retrieval; vep-lit-20261014-eval-pg-czi-unirep

pins

record id: profile-protocol-proteingym-paper-xml-a58110e0; fields: status; attributes.artifact_sha256; sha256: 55a285515bd12e2145721382fccb111bba8ae70f3dd4e1c5aeae2afdc0c00e0d; record id: run-doc-proteingym-readme-md-144fe22b; fields: status; attributes.artifact_sha256; sha256: 70823ddfc9ef5ced58ba8aea340a8d99cf7808ee58b0d2cd4ee1abe5f6684646; record id: use-case-rare-disease-candidate-ranking; fields: status; name; description; facets; attributes.question; attributes.decision; attributes.inputs; attributes.output; attributes.exclusions; attributes.setting; attributes.clinical_scope; attributes.intended_users; sha256: c23eb057b1e65dac505b2a2b94025e4a932c7dcd4ec49c5bf8609c5896ab9aae; record id: vep-lit-20261014-eval-pg-czi-poet-200m; fields: status; links; attributes.origin; attributes.comparison; sha256: ab81207393fad9fa9319890fa0f26559179c6631bb944b6af6a37745d97cd53a; record id: vep-lit-20261014-eval-pg-czi-progen2-base; fields: status; links; attributes.origin; attributes.comparison; sha256: 6ed6562e6a3d93d21af8befbc61b70c9c1595210e88856c59a2897d13c2c5894; record id: vep-lit-20261014-eval-pg-czi-progen2-l; fields: status; links; attributes.origin; attributes.comparison; sha256: c8134e8edbb1d8f510d1d074bc81efbb8a7354f580b035a022a4f6e1d4ef44ab; record id: vep-lit-20261014-eval-pg-czi-progen2-m; fields: status; links; attributes.origin; attributes.comparison; sha256: 89635cd4161f2b179883c75a55ea17c000621e9ed5bf21049c761666032ac97d; record id: vep-lit-20261014-eval-pg-czi-progen2-s; fields: status; links; attributes.origin; attributes.comparison; sha256: 7315617a3b9c8cad80e388494b91cb575ee2bf1efafcdd0cec6ae84499a3ef68; record id: vep-lit-20261014-eval-pg-czi-progen2-xl; fields: status; links; attributes.origin; attributes.comparison; sha256: fe5990bc265dcf08e7aa5edb732075ab9b5dfbd6d36230c5b2b180a2f3e47fe5; record id: vep-lit-20261014-eval-pg-czi-proteingym-clinical-hidden-markov-model; fields: status; links; attributes.origin; attributes.comparison; sha256: 35cf13525b896315655a171884a7f86cc6bcecb8b238cb370bb0e8bdee9b0600; record id: vep-lit-20261014-eval-pg-czi-proteingym-clinical-provean; fields: status; links; attributes.origin; attributes.comparison; sha256: 474a1f1337d8793ff823c2cada1c63ae23f900323607f4c4be1cf9651469534c; record id: vep-lit-20261014-eval-pg-czi-protgpt2; fields: status; links; attributes.origin; attributes.comparison; sha256: 916921f7ab6b29da479fb23c9e4dcd7750381fd362c981e29b1cbb09d5dfce2c; record id: vep-lit-20261014-eval-pg-czi-rita-l; fields: status; links; attributes.origin; attributes.comparison; sha256: a271b12ecb3d5488719b81e9964fb07d00f97de3c863088c895317482a20f35c; record id: vep-lit-20261014-eval-pg-czi-rita-m; fields: status; links; attributes.origin; attributes.comparison; sha256: 8f72e4d9ea18933b0870cda1a943c6e11cc196946bf1edbe12a5425621ca5a04; record id: vep-lit-20261014-eval-pg-czi-rita-s; fields: status; links; attributes.origin; attributes.comparison; sha256: cbf8e9da1065fc4905ee59179617bfe365467d640d367ed0f70c3b489e57381f; record id: vep-lit-20261014-eval-pg-czi-rita-xl; fields: status; links; attributes.origin; attributes.comparison; sha256: 1d3a0444e50cffca15af3f31a1b09a3fa308a34692c0c89f11d33307fdf6cf89; record id: vep-lit-20261014-eval-pg-czi-tranception-l; fields: status; links; attributes.origin; attributes.comparison; sha256: 4c68c5198996e0dc79a634d6f2d84164e128b68d90231ffeffdbe2a1431b7706; record id: vep-lit-20261014-eval-pg-czi-tranception-l-no-retrieval; fields: status; links; attributes.origin; attributes.comparison; sha256: 005dcb9a8e9c5f3b45a5dd95e1089bd2c921a23c09bfa504bfb20d95821b4c04; record id: vep-lit-20261014-eval-pg-czi-tranception-m; fields: status; links; attributes.origin; attributes.comparison; sha256: f289ffd47ec2bf97634c7a62f43fd0eb046a4bf7f25a2808599537b1920e9eaa; record id: vep-lit-20261014-eval-pg-czi-tranception-m-no-retrieval; fields: status; links; attributes.origin; attributes.comparison; sha256: 5f2e4c5da13239982bd1792de7f4badf0575f9ae5ed8b705f4e805ee47bd54de; record id: vep-lit-20261014-eval-pg-czi-unirep; fields: status; links; attributes.origin; attributes.comparison; sha256: 213773c4283e10c24fcce561a616db7b80b51fea9ea1ee4ce64da2bd8aa54127; record id: vep-lit-20261014-protocol-proteingym-clinical-zero-shot-indels-auc-144fe22; fields: status; name; description; facets; source_ids; links; attributes; sha256: ba3046c3d02652ec0647ab047d4523765e3f2566b7c56d88558e35e88ad3a111; record id: vep-lit-20261014-result-pg-czi-poet-200m-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 19a6b5fd8aee6e4e3b25760f2f690d4d201b0c4076329580f9f0321b17e11a0b; record id: vep-lit-20261014-result-pg-czi-progen2-base-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 22bf95ec7dee00d1fd9913533ff1eea2e332ac1867f7dd56696922056e55b90a; record id: vep-lit-20261014-result-pg-czi-progen2-l-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: eae9981116d40294d7024987c32e6c88fac1ce359ecdf0af92b152fce61fbef4; record id: vep-lit-20261014-result-pg-czi-progen2-m-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 4668e5035453b46ae45f0ebb425cca2ad86b48ed0588fae99c6fda7fb9c0a81d; record id: vep-lit-20261014-result-pg-czi-progen2-s-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 21ad15b84ab2e566217ee6ac86eefd681bb69024b8b2857ffaf89e514a8c8041; record id: vep-lit-20261014-result-pg-czi-progen2-xl-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: de9f5e6c3d7f1ab12a6e7dcf0f578863deef1a6235ff55cb8b9ec4568cf581aa; record id: vep-lit-20261014-result-pg-czi-proteingym-clinical-hidden-markov-model-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: d301d439b47b2e58f5a8f481411942f90d4c2c571faf24acc2893727dcf4dc98; record id: vep-lit-20261014-result-pg-czi-proteingym-clinical-provean-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: cfd15be73098d6178201089e825655d04b2eee26e1d1beac0e693afdaea690e0; record id: vep-lit-20261014-result-pg-czi-protgpt2-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: f581d066d25692474664e796e469190dca11d556c0d3f2f5866d83a2d45152d8; record id: vep-lit-20261014-result-pg-czi-rita-l-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: ac85f34542676a7a273251514e6a13ab3b6c82eeb52d5aeea7982637d16d208d; record id: vep-lit-20261014-result-pg-czi-rita-m-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: f2b1531391b279cfd413a2e68f95a52ce03f1eac26364e700ed7688117d202bc; record id: vep-lit-20261014-result-pg-czi-rita-s-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 736924bc60e8220f8cc7c7f3980574a1525544c95f8eb7a7998c682c6e247aa6; record id: vep-lit-20261014-result-pg-czi-rita-xl-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 50c5124b20f2669a7a1ae5f825b824064ac4b636885ff64021fc6aa833f98875; record id: vep-lit-20261014-result-pg-czi-tranception-l-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 9e4b9a44e9785650d8ffba759def1d81cc8371917f349dbe7cddc398e1876e55; record id: vep-lit-20261014-result-pg-czi-tranception-l-no-retrieval-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 8eb48b8a7c9aeacd2d42a058086ec015c8f201eed251856d383456f7efaf5727; record id: vep-lit-20261014-result-pg-czi-tranception-m-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: c2aef2e32880949a9bb281183caa04aef560f50bb48355845c5255019f6af04e; record id: vep-lit-20261014-result-pg-czi-tranception-m-no-retrieval-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 38e7354c7c32f0923a75d94ee9c34d88e3d454efe19477f2b4b7306bb1840296; record id: vep-lit-20261014-result-pg-czi-unirep-average-auc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: faed9cbf69fd14cea5e074201284b674e80bea2598d3d6c0a8a25e2666488f67; record id: vep-lit-20261014-source-proteingym-clinical-zero-shot-indels-auc-summary-144fe22; fields: status; attributes.artifact_sha256; sha256: 1f9e0908fae916b8d568c17399e765d8b6760c7756b8c150545ab1c03e5ce987

• subject: Rank rare-disease variants for review