rewirebio.iobenchmarks
Task

ClinVar 3-prime UTR variant classification

ClinVar variant classification assesses whether zero-shot allele scores distinguish pathogenic from benign annotations.

SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

1 evaluation · 1 result

Overview

Datasets

ClinVar variants meeting a review-status filter, with pathogenic/likely-pathogenic positives and benign/likely-benign negatives; the catalogue retains the 3-prime UTR subset.

Metrics

ROC-based assessment for ClinVar; AUPRC described for the separate imbalanced OMIM comparison must not be imported automatically.

Allowed inputs

Reference/alternate genomic sequences for the 3-prime UTR subset.

SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
Evaluation procedure diagram
How it worksComputational evaluation flow
Computational evaluation flow1. Input: Reference/alternate genomic sequences for the 3-prime UTR subset.. Then: 2. Evaluation: Zero-shot variant scoring; supervised embedding experiments elsewhere are not this protocol.. Then: 3. Readout: ROC-based assessment for ClinVar; AUPRC described for the separate imbalanced OMIM comparison must not be imported automatically.Computational evaluation flow1. Input: Reference/alternate genomic sequences for the 3-prime UTR subset.. Then: 2. Evaluation: Zero-shot variant scoring; supervised embedding experiments elsewhere are not this protocol.. Then: 3. Readout: ROC-based assessment for ClinVar; AUPRC described for the separate imbalanced OMIM comparison must not be imported automatically.Computational evaluation flow1. Input: Reference/alternate genomic sequences for the 3-prime UTR subset.. Then: 2. Evaluation: Zero-shot variant scoring; supervised embedding experiments elsewhere are not this protocol.. Then: 3. Readout: ROC-based assessment for ClinVar; AUPRC described for the separate imbalanced OMIM comparison must not be imported automatically.

Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.

SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

limited source coverage · Automated source review, 2026-09-16. All specifications and missing details

1 recorded evaluation, 1 metric rows. A comparison chart has not yet been validated for these results. The table retains the individual findings and their sources.

View coverage and remaining gaps across all benchmarks

Results

Results are available, but no reviewed comparison panel is linked in this release.

All evaluations

1 evaluation · 1 result. Different protocols are not a single leaderboard.

Filter evaluations

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configurationProtocol and datasetFindingEvidence and details
Configuration: PhyloGPNTask: ClinVar 3-prime UTR variant classification
Dataset: ClinVar 3-prime UTR variants
0.94 AUROC
unitless · unknown

Uncertainty: Not yet extracted

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Superseded
Methods, coverage and source

PhyloGPN: ClinVar 3-prime UTR variant classification

log-likelihood-ratio scoring

Aggregation: Not reported

A Phylogenetic Approach to Genomic Language Modeling · Table 1, 3-prime UTR row, PhyloGPN AUROC column

Source checking is not independent reproduction. Release 2026-11-21-adc15377fab2.

Methods and evaluation design

Procedure, tasks and evaluated configurations

How it works

Evaluation methodology

ClinVar variants meeting a review-status filter, with pathogenic/likely-pathogenic positives and benign/likely-benign negatives; the catalogue retains the 3-prime UTR subset. Zero-shot evaluation, distinct from supervised embedding tasks elsewhere in the paper. ROC-based assessment for ClinVar; AUPRC described for the separate imbalanced OMIM comparison must not be imported automatically. Nucleotide Transformer, HyenaDNA, Caduceus and GPN-MSA comparisons. The cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim.

SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Recorded evaluations

Each evaluation records what was tested and under which conditions.

Run instructions

No runnable recipe has been reviewed for this task. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.

A task describes a biological question. Choose a linked protocol to obtain concrete split and scoring instructions.

Strengths, limitations and unresolved questions

Strengths and limitations

Strengths and considerations

  • The clinical evaluation uses an explicit review-status filter and separates clinical labels from zero-shot scoring.
    SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Limitations and conditions

  • The main ClinVar result uses review-status-filtered variants; it is distinct from the held-out-chromosome ablation. Genome pretraining and downstream label use are separate sources of possible overlap.
    SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
Profile review details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Stable record: reported-task-ed3dd3b83c4505

Specifications

Inputs, training, access and other details

Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
PropertyDescription and evidence
DatasetsClinVar variants meeting a review-status filter, with pathogenic/likely-pathogenic positives and benign/likely-benign negatives; the catalogue retains the 3-prime UTR subset.
SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
SplitsZero-shot evaluation, distinct from supervised embedding tasks elsewhere in the paper.
SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
MetricsROC-based assessment for ClinVar; AUPRC described for the separate imbalanced OMIM comparison must not be imported automatically.
SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
BaselinesNucleotide Transformer, HyenaDNA, Caduceus and GPN-MSA comparisons.
SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
Leakage controlsClinVar variants are selected by review status and pathogenic/benign labels for zero-shot scoring. The main ClinVar experiment does not report exclusion of those loci from genome pretraining. The odd/even chromosome held-out experiment is a separate ablation, not the stated split of the main ClinVar result. · Not reported in inspected sources
SourcesA Phylogenetic Approach to Genomic Language Modeling · Classifying ClinVar variants and Table 1; held-out-human-sequence ablation; cached paragraphs 36,51–53,72
UncertaintyThe cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim. · Not reported in inspected sources
SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
Entity typePaper-specific computational evaluation protocol.
SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
OrganismsHuman ClinVar variants.
SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
AssaysClinical pathogenic/benign assertions meeting a review-status filter.
SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
Allowed inputsReference/alternate genomic sequences for the 3-prime UTR subset.
SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56
AdaptationZero-shot variant scoring; supervised embedding experiments elsewhere are not this protocol.
SourcesA Phylogenetic Approach to Genomic Language Modeling · Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Papers and result coverage

Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.

Paper or primary resourceVersionReference
A Phylogenetic Approach to Genomic Language Modelingpreprint version in PMCRead source
A Phylogenetic Approach to Genomic Language Modelingpreprint version in PMCRead source
Historical gaps recorded on 2026-09-17

The catalogue now holds 1 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.

  • Complete raw matrix acquired. Five methods for assignedUTRrow; othervariantclasses remain separate. Alignment-aware versus sequence-only inputs explicit; no architecture equivalence inferred. Structured extraction pending.
Search and extraction details

source found structured extraction pending

Searches

  • A Phylogenetic Approach to Genomic Language Modeling primary paper benchmark results

Evidence locations

  • Table1 ClinVar3-primeUTR row; likelihood-ratio evaluation

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

20 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-21-adc15377fab2
Property and statementOriginal source and locationReview and provenance
Benchmark research: primary sources
2 values
  • evidence-expansion-p2-phylogpn-2025-807f3a26cbfa
  • phylogpn-2025
Individual claims
A Phylogenetic Approach to Genomic Language Modeling

Original source ↗

data/entities/sources.jsonl, records phylogpn-2025 and evidence-expansion-p2-phylogpn-2025-807f3a26cbfa, both with artifact_sha256 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Version: preprint version in PMC
Retrieved: 2026-09-16T10:38:57.558218+00:00

conflicting claim

["ai-assisted-source-review","scope-review"] · 2026-11-16T03:00:00Z

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

primary_sources named only the retired duplicate. It now names phylogpn-2025, the same bytes under the surviving ID. phylogpn-2025 and evidence-expansion-p2-phylogpn-2025-807f3a26cbfa pin the same bytes: both carry artifact_sha256 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224 and the same Europe PMC fullTextXML artifact URL. docs/reviews/2026-11-16-phylogpn.md ruling 3 made phylogpn-2025 the survivor, gave it a supersedes link to the duplicate and set the duplicate's status to superseded, and left this record's references to the coordinator. This task was the only record still citing the duplicate. Retargeting the reference withdraws no evidence, because the two records pin one artifact, and moves no locator. The record stays needs_review and the other three legacy records behind b2-phylogpn-2025 were left alone: ruling 5 is right that retiring that silo is a decision of its own.

Conflicting source claim: "[\"phylogpn-2025\"]". Claim record status: source_checked.

Field: attributes.benchmark_research.primary_sources

Claim: metadata-correction-coordfix-20261116-legacy-task-primary-sources

Source artifact SHA-256: 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Diagram caption
Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.
Individual claims
A Phylogenetic Approach to Genomic Language Modeling

Original source ↗

Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Version: preprint version in PMC
Retrieved: 2026-09-16T10:38:57.558218+00:00

source checked

automated source review · 2026-09-16

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Field: attributes.profile.diagram.caption

Source artifact SHA-256: 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Diagram steps
  • Input: Reference/alternate genomic sequences for the 3-prime UTR subset.
  • Evaluation: Zero-shot variant scoring; supervised embedding experiments elsewhere are not this protocol.
  • Readout: ROC-based assessment for ClinVar; AUPRC described for the separate imbalanced OMIM comparison must not be imported automatically.
Individual claims
A Phylogenetic Approach to Genomic Language Modeling

Original source ↗

Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Version: preprint version in PMC
Retrieved: 2026-09-16T10:38:57.558218+00:00

source checked

automated source review · 2026-09-16

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Field: attributes.profile.diagram.steps

Source artifact SHA-256: 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Diagram title
Computational evaluation flow
Individual claims
A Phylogenetic Approach to Genomic Language Modeling

Original source ↗

Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Version: preprint version in PMC
Retrieved: 2026-09-16T10:38:57.558218+00:00

source checked

automated source review · 2026-09-16

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Field: attributes.profile.diagram.title

Source artifact SHA-256: 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Datasets
ClinVar variants meeting a review-status filter, with pathogenic/likely-pathogenic positives and benign/likely-benign negatives; the catalogue retains the 3-prime UTR subset.
Individual claims
A Phylogenetic Approach to Genomic Language Modeling

Original source ↗

Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Version: preprint version in PMC
Retrieved: 2026-09-16T10:38:57.558218+00:00

source checked

automated source review · 2026-09-16

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Field: attributes.profile.facts.0.value

Source artifact SHA-256: 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Splits
Zero-shot evaluation, distinct from supervised embedding tasks elsewhere in the paper.
Individual claims
A Phylogenetic Approach to Genomic Language Modeling

Original source ↗

Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Version: preprint version in PMC
Retrieved: 2026-09-16T10:38:57.558218+00:00

source checked

automated source review · 2026-09-16

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Field: attributes.profile.facts.1.value

Source artifact SHA-256: 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Adaptation
Zero-shot variant scoring; supervised embedding experiments elsewhere are not this protocol.
Individual claims
A Phylogenetic Approach to Genomic Language Modeling

Original source ↗

Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Version: preprint version in PMC
Retrieved: 2026-09-16T10:38:57.558218+00:00

source checked

automated source review · 2026-09-16

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Field: attributes.profile.facts.10.value

Source artifact SHA-256: 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Metrics
ROC-based assessment for ClinVar; AUPRC described for the separate imbalanced OMIM comparison must not be imported automatically.
Individual claims
A Phylogenetic Approach to Genomic Language Modeling

Original source ↗

Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Version: preprint version in PMC
Retrieved: 2026-09-16T10:38:57.558218+00:00

source checked

automated source review · 2026-09-16

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Field: attributes.profile.facts.2.value

Source artifact SHA-256: 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Baselines
Nucleotide Transformer, HyenaDNA, Caduceus and GPN-MSA comparisons.
Individual claims
A Phylogenetic Approach to Genomic Language Modeling

Original source ↗

Results: Log Likelihood Ratio Evaluation; ClinVar and OMIM tests; cached text lines 46–48, 52, 56

Version: preprint version in PMC
Retrieved: 2026-09-16T10:38:57.558218+00:00

source checked

automated source review · 2026-09-16

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Field: attributes.profile.facts.3.value

Source artifact SHA-256: 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Leakage controls
ClinVar variants are selected by review status and pathogenic/benign labels for zero-shot scoring. The main ClinVar experiment does not report exclusion of those loci from genome pretraining. The odd/even chromosome held-out experiment is a separate ablation, not the stated split of the main ClinVar result.
Individual claims
A Phylogenetic Approach to Genomic Language Modeling

Original source ↗

Classifying ClinVar variants and Table 1; held-out-human-sequence ablation; cached paragraphs 36,51–53,72

Version: preprint version in PMC
Retrieved: 2026-09-16T10:38:57.558218+00:00

unreported

automated source review · 2026-09-16

Source has a recorded evidence concern. Consult its source page before using the claim.

Audit details

Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.

Field: attributes.profile.facts.4.value

Source artifact SHA-256: 807f3a26cbfa9b5ce238d92164bd523302c67d1c5794b08273c51cca1acd4224

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-21-adc15377fab2 · Record review: needs review

2 source records and release historyDownload this release (gzip)
Technical metadata and record identifiers

Technical metadata for this record is tracked under stable ID reported-task-ed3dd3b83c4505.

Related records

Suggest a correction