Evidence claim

Relevance of GFMBench-API TraitGym Mendelian, zs-mlm-llr to "Select regulatory variants and genes for functional follow-up"

Use case: Select regulatory variants and genes for functional follow-up
Assessed by: GFMBench-API TraitGym Mendelian, zs-mlm-llr
Relevance: Proxy evidence
Endpoint: Zero-shot AUROC and AUPRC of masked log-likelihood ratio for masked genomic language models (DNA-BERT, DNABERT-2, NTv3, Caduceus-Ps) on TraitGym Mendelian

GFMBench-API zero-shot masked log-likelihood ratio scores on TraitGym Mendelian as evidence for choosing noncoding variants to edit.

4 reviewed evaluations

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

19 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-22-1b7cf027e4f6
Property and statement	Original source and location	Review and provenance
Citation locations 1 values source id lit-genomics-20261011-gfmbench-source-larey2026 locator Table 4, task 'TraitGym Mendelian'; appendix task description Context-only references	GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models Original source ↗ lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description Version: bioRxiv preprint version 2, posted 2026-08-11 Retrieved: 2026-10-11T00:44:58Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.citation_locators` Source artifact SHA-256: `df9934c88ee245b1e19e155174169d6b6da57ae07c1a04bbea7115423facdf72` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Comparison group gfmbench-traitgym Context-only references	GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models Original source ↗ lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description Version: bioRxiv preprint version 2, posted 2026-08-11 Retrieved: 2026-10-11T00:44:58Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.comparison_group` Source artifact SHA-256: `df9934c88ee245b1e19e155174169d6b6da57ae07c1a04bbea7115423facdf72` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Comparison title GFMBench-API zero-shot scores on TraitGym Context-only references	GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models Original source ↗ lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description Version: bioRxiv preprint version 2, posted 2026-08-11 Retrieved: 2026-10-11T00:44:58Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.comparison_title` Source artifact SHA-256: `df9934c88ee245b1e19e155174169d6b6da57ae07c1a04bbea7115423facdf72` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Constraints 3 values Inspect every linked evaluation's source locator before citing a result. Do not combine this mapping's evaluations with any other protocol's results. Compare only with other rows of the same GFMBench-API table; TraitGym's own paper uses different context lengths and scoring. Context-only references	GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models Original source ↗ lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description Version: bioRxiv preprint version 2, posted 2026-08-11 Retrieved: 2026-10-11T00:44:58Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.constraints` Source artifact SHA-256: `df9934c88ee245b1e19e155174169d6b6da57ae07c1a04bbea7115423facdf72` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Endpoint Zero-shot AUROC and AUPRC of masked log-likelihood ratio for masked genomic language models (DNA-BERT, DNABERT-2, NTv3, Caduceus-Ps) on TraitGym Mendelian Context-only references	GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models Original source ↗ lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description Version: bioRxiv preprint version 2, posted 2026-08-11 Retrieved: 2026-10-11T00:44:58Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.endpoint` Source artifact SHA-256: `df9934c88ee245b1e19e155174169d6b6da57ae07c1a04bbea7115423facdf72` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Claimed field links:assessed_by:lit-genomics-20261011-gfmbench-protocol-traitgym-mendelian-zs-mlm-llr Context-only references	GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models Original source ↗ lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description Version: bioRxiv preprint version 2, posted 2026-08-11 Retrieved: 2026-10-11T00:44:58Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.field` Source artifact SHA-256: `df9934c88ee245b1e19e155174169d6b6da57ae07c1a04bbea7115423facdf72` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Headline metric auroc Context-only references	GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models Original source ↗ lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description Version: bioRxiv preprint version 2, posted 2026-08-11 Retrieved: 2026-10-11T00:44:58Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.headline_metric` Source artifact SHA-256: `df9934c88ee245b1e19e155174169d6b6da57ae07c1a04bbea7115423facdf72` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Limitations 3 values Preprint, not peer reviewed. Mendelian AUROCs are all 0.47-0.52, close to 0.5; no uncertainty is printed. Curated Mendelian regulatory variants, not experimental allele effects. Context-only references	GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models Original source ↗ lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description Version: bioRxiv preprint version 2, posted 2026-08-11 Retrieved: 2026-10-11T00:44:58Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.limitations` Source artifact SHA-256: `df9934c88ee245b1e19e155174169d6b6da57ae07c1a04bbea7115423facdf72` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Pinned records 15 values record id lit-genomics-20261011-gfmbench-eval-caduceus-ps-zero-shot-traitgym-mendelian-zs-mlm-llr fields status links attributes.origin attributes.comparison sha256 2f0fdb18b6df3308704d9c9686829bdc255ee203ee3703826ffe6e6b60784fe4 record id lit-genomics-20261011-gfmbench-eval-dna-bert-zero-shot-traitgym-mendelian-zs-mlm-llr fields status links attributes.origin attributes.comparison sha256 14cee4dbce865faadd358fce84c398c3c741f55622eaebde3e9026c3674b6cc5 record id lit-genomics-20261011-gfmbench-eval-dnabert-2-zero-shot-traitgym-mendelian-zs-mlm-llr fields status links attributes.origin attributes.comparison sha256 69bae7780b5cdeaec20fdac09e7f2b4e95d06b252875e8cec31226b003fc8655 record id lit-genomics-20261011-gfmbench-eval-ntv3-zero-shot-traitgym-mendelian-zs-mlm-llr fields status links attributes.origin attributes.comparison sha256 853d20afc3ecdf468496a9735a7fbcfeb3e01446e0a77aed2087b33c6c686a9c record id lit-genomics-20261011-gfmbench-protocol-traitgym-mendelian-zs-mlm-llr fields status name description facets source_ids links attributes sha256 63b7a5f95d610fac9fceda0e52cf0675a9c83d373cf45958472781e5fe7ee6a3 record id lit-genomics-20261011-gfmbench-result-caduceus-ps-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-prc fields status attributes.metric attributes.metric_qualifier attributes.printed_value attributes.numeric_value sha256 6efd66fcd138a0157bb8806ee2db06ce809766f049f1924b6dad7fa5d9e2c830 record id lit-genomics-20261011-gfmbench-result-caduceus-ps-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-roc fields status attributes.metric attributes.metric_qualifier attributes.printed_value attributes.numeric_value sha256 c145990091aa9db90fe73bd35927adcea9baaeb61eb8257ccfcc0bd60eea102a record id lit-genomics-20261011-gfmbench-result-dna-bert-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-prc fields status attributes.metric attributes.metric_qualifier attributes.printed_value attributes.numeric_value sha256 3ce7eee2134e5ac3e35b0ea78d73c2f8813cb57010620daf20f1726e3514f822 record id lit-genomics-20261011-gfmbench-result-dna-bert-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-roc fields status attributes.metric attributes.metric_qualifier attributes.printed_value attributes.numeric_value sha256 a11266f902e78891ad67896e1dbff80b3060d4d4db9767b7906d971a198d52e4 record id lit-genomics-20261011-gfmbench-result-dnabert-2-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-prc fields status attributes.metric attributes.metric_qualifier attributes.printed_value attributes.numeric_value sha256 8f52a343584d34b1e0996ed426db6899e79cf2361fa2ea656d46162a6b1c1d6e record id lit-genomics-20261011-gfmbench-result-dnabert-2-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-roc fields status attributes.metric attributes.metric_qualifier attributes.printed_value attributes.numeric_value sha256 f643af62fef1d4ed9fee8c5e7b327c299db0c2349df5581e2ed15fcc10b5235a record id lit-genomics-20261011-gfmbench-result-ntv3-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-prc fields status attributes.metric attributes.metric_qualifier attributes.printed_value attributes.numeric_value sha256 8a87dcd148ed1be901cb6c8732e3b510bb5f6fee93ab28f9ebea9fbcd65f9c4a record id lit-genomics-20261011-gfmbench-result-ntv3-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-roc fields status attributes.metric attributes.metric_qualifier attributes.printed_value attributes.numeric_value sha256 c86c080478e623fc1951266a89614138ac12a1ea070f57c6c30f25a6e361edf9 record id lit-genomics-20261011-gfmbench-source-larey2026 fields status attributes.artifact_sha256 sha256 a461e55e4db9e70efca0dd40a686ae239e2e254757de72623099c721fd6eb0fc record id use-case-regulatory-variant-gene-follow-up fields status name description facets attributes.question attributes.decision attributes.inputs attributes.output attributes.exclusions attributes.setting attributes.clinical_scope attributes.intended_users sha256 0071547e01ccaec30238aea9e4882dd92e6f51be9c967c7fb88bc61d2788b870 Context-only references	GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models Original source ↗ lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description Version: bioRxiv preprint version 2, posted 2026-08-11 Retrieved: 2026-10-11T00:44:58Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.pins` Source artifact SHA-256: `df9934c88ee245b1e19e155174169d6b6da57ae07c1a04bbea7115423facdf72` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Rationale Proxy, not direct: the protocol ranks candidate noncoding variants by zero-shot model scores, which matches the first step of choosing allele edits, but it gives no element-gene link and no effect direction, and it is scored on the GFMBench-API re-implementation of TraitGym rather than on perturbation data. Context-only references	GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models Original source ↗ lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description Version: bioRxiv preprint version 2, posted 2026-08-11 Retrieved: 2026-10-11T00:44:58Z	not individually reviewed No individual claim review recorded Audit details Field: `attributes.rationale` Source artifact SHA-256: `df9934c88ee245b1e19e155174169d6b6da57ae07c1a04bbea7115423facdf72` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact

How to read the evidence

Sources and history

Release 2026-10-22-1b7cf027e4f6 · Record review: source checked

1 source record and release history

GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models · Original source · bioRxiv preprint version 2, posted 2026-08-11

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: use-case-mapping-lit-genomics-20261011-gfmbench-traitgym-mendelian-zs-mlm-llr

field: links:assessed_by:lit-genomics-20261011-gfmbench-protocol-traitgym-mendelian-zs-mlm-llr
value: lit-genomics-20261011-gfmbench-protocol-traitgym-mendelian-zs-mlm-llr
relevance: proxy
endpoint: Zero-shot AUROC and AUPRC of masked log-likelihood ratio for masked genomic language models (DNA-BERT, DNABERT-2, NTv3, Caduceus-Ps) on TraitGym Mendelian
rationale: Proxy, not direct: the protocol ranks candidate noncoding variants by zero-shot model scores, which matches the first step of choosing allele edits, but it gives no element-gene link and no effect direction, and it is scored on the GFMBench-API re-implementation of TraitGym rather than on perturbation data.
constraints: Inspect every linked evaluation's source locator before citing a result.; Do not combine this mapping's evaluations with any other protocol's results.; Compare only with other rows of the same GFMBench-API table; TraitGym's own paper uses different context lengths and scoring.
limitations: Preprint, not peer reviewed.; Mendelian AUROCs are all 0.47-0.52, close to 0.5; no uncertainty is printed.; Curated Mendelian regulatory variants, not experimental allele effects.
citation locators: source id: lit-genomics-20261011-gfmbench-source-larey2026; locator: Table 4, task 'TraitGym Mendelian'; appendix task description
source locator: lit-genomics-20261011-gfmbench-source-larey2026: Table 4, task 'TraitGym Mendelian'; appendix task description
headline metric: auroc
reason: Add a relevance judgement for a genomic foundation model benchmark protocol from the 2026-10-11 genomics suite pass, reviewed in docs/reviews/2026-10-11-genomics-gfm-suites.md.
comparison group: gfmbench-traitgym
comparison title: GFMBench-API zero-shot scores on TraitGym
stratum label: TraitGym Mendelian, masked log-likelihood ratio
stratum order: 2
reviewed evaluations: lit-genomics-20261011-gfmbench-eval-caduceus-ps-zero-shot-traitgym-mendelian-zs-mlm-llr; lit-genomics-20261011-gfmbench-eval-dna-bert-zero-shot-traitgym-mendelian-zs-mlm-llr; lit-genomics-20261011-gfmbench-eval-dnabert-2-zero-shot-traitgym-mendelian-zs-mlm-llr; lit-genomics-20261011-gfmbench-eval-ntv3-zero-shot-traitgym-mendelian-zs-mlm-llr
review: method: ai-assisted-source-review; scope-review; method note: Reviewer checked the endpoint against the use case's inputs and output, the proxy rating and the limitations; every evaluation on the protocol had its values checked in the batch review. Corrections listed in review.md were applied by the extractor.; reviewer: claude; reviewer note: Claude review agent reviewer-gen-3, independent of the extractor lit-genomics; no human review claimed; reviewed at: 2026-10-11T14:40:00Z; note: Automated review; not human scientific review.
pins: record id: lit-genomics-20261011-gfmbench-eval-caduceus-ps-zero-shot-traitgym-mendelian-zs-mlm-llr; fields: status; links; attributes.origin; attributes.comparison; sha256: 2f0fdb18b6df3308704d9c9686829bdc255ee203ee3703826ffe6e6b60784fe4; record id: lit-genomics-20261011-gfmbench-eval-dna-bert-zero-shot-traitgym-mendelian-zs-mlm-llr; fields: status; links; attributes.origin; attributes.comparison; sha256: 14cee4dbce865faadd358fce84c398c3c741f55622eaebde3e9026c3674b6cc5; record id: lit-genomics-20261011-gfmbench-eval-dnabert-2-zero-shot-traitgym-mendelian-zs-mlm-llr; fields: status; links; attributes.origin; attributes.comparison; sha256: 69bae7780b5cdeaec20fdac09e7f2b4e95d06b252875e8cec31226b003fc8655; record id: lit-genomics-20261011-gfmbench-eval-ntv3-zero-shot-traitgym-mendelian-zs-mlm-llr; fields: status; links; attributes.origin; attributes.comparison; sha256: 853d20afc3ecdf468496a9735a7fbcfeb3e01446e0a77aed2087b33c6c686a9c; record id: lit-genomics-20261011-gfmbench-protocol-traitgym-mendelian-zs-mlm-llr; fields: status; name; description; facets; source_ids; links; attributes; sha256: 63b7a5f95d610fac9fceda0e52cf0675a9c83d373cf45958472781e5fe7ee6a3; record id: lit-genomics-20261011-gfmbench-result-caduceus-ps-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-prc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 6efd66fcd138a0157bb8806ee2db06ce809766f049f1924b6dad7fa5d9e2c830; record id: lit-genomics-20261011-gfmbench-result-caduceus-ps-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-roc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: c145990091aa9db90fe73bd35927adcea9baaeb61eb8257ccfcc0bd60eea102a; record id: lit-genomics-20261011-gfmbench-result-dna-bert-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-prc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 3ce7eee2134e5ac3e35b0ea78d73c2f8813cb57010620daf20f1726e3514f822; record id: lit-genomics-20261011-gfmbench-result-dna-bert-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-roc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: a11266f902e78891ad67896e1dbff80b3060d4d4db9767b7906d971a198d52e4; record id: lit-genomics-20261011-gfmbench-result-dnabert-2-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-prc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 8f52a343584d34b1e0996ed426db6899e79cf2361fa2ea656d46162a6b1c1d6e; record id: lit-genomics-20261011-gfmbench-result-dnabert-2-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-roc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: f643af62fef1d4ed9fee8c5e7b327c299db0c2349df5581e2ed15fcc10b5235a; record id: lit-genomics-20261011-gfmbench-result-ntv3-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-prc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 8a87dcd148ed1be901cb6c8732e3b510bb5f6fee93ab28f9ebea9fbcd65f9c4a; record id: lit-genomics-20261011-gfmbench-result-ntv3-zero-shot-traitgym-mendelian-zs-mlm-llr-msk-llr-roc; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: c86c080478e623fc1951266a89614138ac12a1ea070f57c6c30f25a6e361edf9; record id: lit-genomics-20261011-gfmbench-source-larey2026; fields: status; attributes.artifact_sha256; sha256: a461e55e4db9e70efca0dd40a686ae239e2e254757de72623099c721fd6eb0fc; record id: use-case-regulatory-variant-gene-follow-up; fields: status; name; description; facets; attributes.question; attributes.decision; attributes.inputs; attributes.output; attributes.exclusions; attributes.setting; attributes.clinical_scope; attributes.intended_users; sha256: 0071547e01ccaec30238aea9e4882dd92e6f51be9c967c7fb88bc61d2788b870

Related records

subject: Select regulatory variants and genes for functional follow-up

Suggest a correction