rewirebio.iobenchmarks
Task

protein variant-effect classification

Structure-informed variant-effect evaluation compares ranking and binary classification against experimental protein measurements.

SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

1 evaluation · 1 result

Overview

Datasets

DeepSequence-derived missense-variant DMS datasets, excluding tRNA and viral-family sets; available family sequences and structures support adaptation.

Metrics

Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants.

Allowed inputs

Protein sequence; available family sequences/structures are used during adaptation.

SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
Evaluation procedure diagram
How it worksComputational evaluation flow
Computational evaluation flow1. Input: Protein sequence; available family sequences/structures are used during adaptation.. Then: 2. Evaluation: Unsupervised family adaptation plus labeled-validation model selection; no fitness-label fitting of model parameters.. Then: 3. Readout: Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants.Computational evaluation flow1. Input: Protein sequence; available family sequences/structures are used during adaptation.. Then: 2. Evaluation: Unsupervised family adaptation plus labeled-validation model selection; no fitness-label fitting of model parameters.. Then: 3. Readout: Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants.Computational evaluation flow1. Input: Protein sequence; available family sequences/structures are used during adaptation.. Then: 2. Evaluation: Unsupervised family adaptation plus labeled-validation model selection; no fitness-label fitting of model parameters.. Then: 3. Readout: Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants.

Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.

SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

limited source coverage · Automated source review, 2026-09-16. All specifications and missing details

1 recorded evaluation, 1 metric rows. A comparison chart has not yet been validated for these results. The table retains the individual findings and their sources.

View coverage and remaining gaps across all benchmarks

Results

Results are available, but no reviewed comparison panel is linked in this release.

All evaluations

1 evaluation · 1 result. Different protocols are not a single leaderboard.

Filter evaluations

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configurationProtocol and datasetFindingEvidence and details
Configuration: structure-informed pLMTask: protein variant-effect classification
Dataset: variant-effects benchmark
0.803 AUROC
unitless · unknown

Uncertainty: Not yet extracted

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

structure-informed pLM: protein variant-effect classification

combined amino-acid, secondary structure, solvent accessibility and contact-map scoring

Aggregation: Not reported

Structure-Informed Protein Language Models are Robust Predictors for Variant Effects · PMC12068927 HTML, Table4, AA+SS+RSA+CM row, AUROC column

Source checking is not independent reproduction. Release 2026-11-21-adc15377fab2.

Methods and evaluation design

Procedure, tasks and evaluated configurations

How it works

Evaluation methodology

DeepSequence-derived missense-variant DMS datasets, excluding tRNA and viral-family sets; available family sequences and structures support adaptation. A labeled validation subset selects the structural weighting and model; labels are not used to fit the model parameters. The exact held-out membership after selection remains unextracted. Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants. PSSM, EVmutation, DeepSequence, Wavenet and several protein language models; competing scores are obtained from the ESM-1v repository. Family-specific adaptation and validation-label model selection are part of the protocol; it should not be represented as selection without labeled data. One-sided paired Wilcoxon signed-rank comparisons across assay performances are reported for the main method comparison.

SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Recorded evaluations

Each evaluation records what was tested and under which conditions.

Run instructions

No runnable recipe has been reviewed for this task. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.

A task describes a biological question. Choose a linked protocol to obtain concrete split and scoring instructions.

Strengths, limitations and unresolved questions

Strengths and limitations

Profile review details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Stable record: reported-task-83be0998084c91

Specifications

Inputs, training, access and other details

Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
PropertyDescription and evidence
DatasetsDeepSequence-derived missense-variant DMS datasets, excluding tRNA and viral-family sets; available family sequences and structures support adaptation.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
SplitsA labeled validation subset selects the structural weighting and model; labels are not used to fit the model parameters. The exact held-out membership after selection remains unextracted.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
MetricsSpearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
BaselinesPSSM, EVmutation, DeepSequence, Wavenet and several protein language models; competing scores are obtained from the ESM-1v repository.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
Leakage controlsFamily-specific adaptation and validation-label model selection are part of the protocol; it should not be represented as selection without labeled data.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
UncertaintyOne-sided paired Wilcoxon signed-rank comparisons across assay performances are reported for the main method comparison.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
Entity typePaper-specific computational evaluation protocol.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
OrganismsNonviral protein families in the selected DMS collection.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
AssaysDMS missense-variant measurements.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
Allowed inputsProtein sequence; available family sequences/structures are used during adaptation.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
AdaptationUnsupervised family adaptation plus labeled-validation model selection; no fitness-label fitting of model parameters.
SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Papers and result coverage

Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.

Paper or primary resourceVersionReference
Structure-informed protein language models are robust predictors for variant effects - PMCHuman Genetics 2025 journal article (online 2024)Read source
Historical gaps recorded on 2026-09-17

The catalogue now holds 1 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.

  • Artifact returned HTML rather than XML, so XML table parser intentionally rejected it. Prior reviewed observation retained. Complete table extraction and source-version comparison remain pending.
Search and extraction details

source found structured extraction pending

Searches

  • Structure-Informed Protein Language Models are Robust Predictors for Variant Effects primary paper benchmark results

Evidence locations

  • Primary publisher/PubMedCentral HTML; VEP result tables

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

18 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-21-adc15377fab2
Property and statementOriginal source and locationReview and provenance
Diagram caption
Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.
Individual claims
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot)

Original source ↗

Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Version: Human Genetics 2025 journal article (online 2024)
Retrieved: 2026-09-16T19:58:11.209175+00:00

source checked

automated source review · 2026-09-16

Audit details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Field: attributes.profile.diagram.caption

Source artifact SHA-256: 6686d2646e7b1b1203a7ef6d48bacf966b4cd6b0895fd2855551937c6bb87111

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Diagram steps
  • Input: Protein sequence; available family sequences/structures are used during adaptation.
  • Evaluation: Unsupervised family adaptation plus labeled-validation model selection; no fitness-label fitting of model parameters.
  • Readout: Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants.
Individual claims
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot)

Original source ↗

Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Version: Human Genetics 2025 journal article (online 2024)
Retrieved: 2026-09-16T19:58:11.209175+00:00

source checked

automated source review · 2026-09-16

Audit details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Field: attributes.profile.diagram.steps

Source artifact SHA-256: 6686d2646e7b1b1203a7ef6d48bacf966b4cd6b0895fd2855551937c6bb87111

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Diagram title
Computational evaluation flow
Individual claims
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot)

Original source ↗

Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Version: Human Genetics 2025 journal article (online 2024)
Retrieved: 2026-09-16T19:58:11.209175+00:00

source checked

automated source review · 2026-09-16

Audit details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Field: attributes.profile.diagram.title

Source artifact SHA-256: 6686d2646e7b1b1203a7ef6d48bacf966b4cd6b0895fd2855551937c6bb87111

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Datasets
DeepSequence-derived missense-variant DMS datasets, excluding tRNA and viral-family sets; available family sequences and structures support adaptation.
Individual claims
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot)

Original source ↗

Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Version: Human Genetics 2025 journal article (online 2024)
Retrieved: 2026-09-16T19:58:11.209175+00:00

source checked

automated source review · 2026-09-16

Audit details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Field: attributes.profile.facts.0.value

Source artifact SHA-256: 6686d2646e7b1b1203a7ef6d48bacf966b4cd6b0895fd2855551937c6bb87111

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Splits
A labeled validation subset selects the structural weighting and model; labels are not used to fit the model parameters. The exact held-out membership after selection remains unextracted.
Individual claims
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot)

Original source ↗

Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Version: Human Genetics 2025 journal article (online 2024)
Retrieved: 2026-09-16T19:58:11.209175+00:00

source checked

automated source review · 2026-09-16

Audit details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Field: attributes.profile.facts.1.value

Source artifact SHA-256: 6686d2646e7b1b1203a7ef6d48bacf966b4cd6b0895fd2855551937c6bb87111

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Adaptation
Unsupervised family adaptation plus labeled-validation model selection; no fitness-label fitting of model parameters.
Individual claims
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot)

Original source ↗

Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Version: Human Genetics 2025 journal article (online 2024)
Retrieved: 2026-09-16T19:58:11.209175+00:00

source checked

automated source review · 2026-09-16

Audit details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Field: attributes.profile.facts.10.value

Source artifact SHA-256: 6686d2646e7b1b1203a7ef6d48bacf966b4cd6b0895fd2855551937c6bb87111

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Metrics
Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants.
Individual claims
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot)

Original source ↗

Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Version: Human Genetics 2025 journal article (online 2024)
Retrieved: 2026-09-16T19:58:11.209175+00:00

source checked

automated source review · 2026-09-16

Audit details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Field: attributes.profile.facts.2.value

Source artifact SHA-256: 6686d2646e7b1b1203a7ef6d48bacf966b4cd6b0895fd2855551937c6bb87111

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Baselines
PSSM, EVmutation, DeepSequence, Wavenet and several protein language models; competing scores are obtained from the ESM-1v repository.
Individual claims
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot)

Original source ↗

Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Version: Human Genetics 2025 journal article (online 2024)
Retrieved: 2026-09-16T19:58:11.209175+00:00

source checked

automated source review · 2026-09-16

Audit details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Field: attributes.profile.facts.3.value

Source artifact SHA-256: 6686d2646e7b1b1203a7ef6d48bacf966b4cd6b0895fd2855551937c6bb87111

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Leakage controls
Family-specific adaptation and validation-label model selection are part of the protocol; it should not be represented as selection without labeled data.
Individual claims
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot)

Original source ↗

Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Version: Human Genetics 2025 journal article (online 2024)
Retrieved: 2026-09-16T19:58:11.209175+00:00

source checked

automated source review · 2026-09-16

Audit details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Field: attributes.profile.facts.4.value

Source artifact SHA-256: 6686d2646e7b1b1203a7ef6d48bacf966b4cd6b0895fd2855551937c6bb87111

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Uncertainty
One-sided paired Wilcoxon signed-rank comparisons across assay performances are reported for the main method comparison.
Individual claims
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot)

Original source ↗

Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods

Version: Human Genetics 2025 journal article (online 2024)
Retrieved: 2026-09-16T19:58:11.209175+00:00

source checked

automated source review · 2026-09-16

Audit details

Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.

Field: attributes.profile.facts.5.value

Source artifact SHA-256: 6686d2646e7b1b1203a7ef6d48bacf966b4cd6b0895fd2855551937c6bb87111

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-21-adc15377fab2 · Record review: needs review

3 source records and release historyDownload this release (gzip)
Technical metadata and record identifiers

Technical metadata for this record is tracked under stable ID reported-task-83be0998084c91.

Related records

Suggest a correction