Datasets
DeepSequence-derived missense-variant DMS datasets, excluding tRNA and viral-family sets; available family sequences and structures support adaptation.
Task
protein variant-effect classification
Structure-informed variant-effect evaluation compares ranking and binary classification against experimental protein measurements.
Sources
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methodsOverview
Metrics
Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants.
Allowed inputs
Protein sequence; available family sequences/structures are used during adaptation.
Sources
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methodsEvaluation procedure diagram
How it worksComputational evaluation flow
Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.
Sources
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methodslimited source coverage · Automated source review, 2026-09-16. All specifications and missing details
Results
Results are available, but no reviewed comparison panel is linked in this release.
All evaluations
1 evaluation · 1 result. Different protocols are not a single leaderboard.
Filter evaluations
Applied filters: All linked evaluations
| Tested configuration | Protocol and dataset | Finding | Evidence and details |
|---|---|---|---|
| Configuration: structure-informed pLM | Task: protein variant-effect classification Dataset: variant-effects benchmark | 0.803 AUROC fraction · unknown Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourcestructure-informed pLM: protein variant-effect classification combined amino-acid, secondary structure, solvent accessibility and contact-map scoring Aggregation: Not reported Structure-Informed Protein Language Models are Robust Predictors for Variant Effects · PMC12068927 HTML, Table4, AA+SS+RSA+CM row, AUROC column |
Source checking is not independent reproduction. Release 2026-09-29-06401fd5b220.
Methods and evaluation design
Procedure, tasks and evaluated configurations
How it works
Evaluation methodology
DeepSequence-derived missense-variant DMS datasets, excluding tRNA and viral-family sets; available family sequences and structures support adaptation. A labeled validation subset selects the structural weighting and model; labels are not used to fit the model parameters. The exact held-out membership after selection remains unextracted. Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants. PSSM, EVmutation, DeepSequence, Wavenet and several protein language models; competing scores are obtained from the ESM-1v repository. Family-specific adaptation and validation-label model selection are part of the protocol; it should not be represented as selection without labeled data. One-sided paired Wilcoxon signed-rank comparisons across assay performances are reported for the main method comparison.
Sources
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methodsRecorded evaluations
Each evaluation records what was tested and under which conditions.
Run instructions
No runnable recipe has been reviewed for this task. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.
A task describes a biological question. Choose a linked protocol to obtain concrete split and scoring instructions.
Strengths, limitations and unresolved questions
Strengths and limitations
Strengths and considerations
- The paper explicitly distinguishes parameter fitting from validation-label model selection.
Sources
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
Limitations and conditions
- The final journal Table 4 scoring variant is distinct from the main Table 2 configuration. Exact test membership, scoring configuration and source version must remain attached to each reported result.
Sources
Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods
Profile review details
Task-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced.
Stable record: reported-task-83be0998084c91Specifications
Inputs, training, access and other details
Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Datasets | DeepSequence-derived missense-variant DMS datasets, excluding tRNA and viral-family sets; available family sequences and structures support adaptation.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
| Splits | A labeled validation subset selects the structural weighting and model; labels are not used to fit the model parameters. The exact held-out membership after selection remains unextracted.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
| Metrics | Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
| Baselines | PSSM, EVmutation, DeepSequence, Wavenet and several protein language models; competing scores are obtained from the ESM-1v repository.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
| Leakage controls | Family-specific adaptation and validation-label model selection are part of the protocol; it should not be represented as selection without labeled data.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
| Uncertainty | One-sided paired Wilcoxon signed-rank comparisons across assay performances are reported for the main method comparison.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
| Entity type | Paper-specific computational evaluation protocol.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
| Organisms | Nonviral protein families in the selected DMS collection.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
| Assays | DMS missense-variant measurements.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
| Allowed inputs | Protein sequence; available family sequences/structures are used during adaptation.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
| Adaptation | Unsupervised family adaptation plus labeled-validation model selection; no fitness-label fitting of model parameters.SourcesStructure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods |
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Papers and result coverage
Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
| Paper or primary resource | Version | Reference |
|---|---|---|
| Structure-informed protein language models are robust predictors for variant effects - PMC | Human Genetics 2025 journal article (online 2024) | Read source |
Historical gaps recorded on 2026-09-17
The catalogue now holds 1 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.
- Artifact returned HTML rather than XML, so XML table parser intentionally rejected it. Prior reviewed observation retained. Complete table extraction and source-version comparison remain pending.
Search and extraction details
source found structured extraction pending
Searches
- Structure-Informed Protein Language Models are Robust Predictors for Variant Effects primary paper benchmark results
Evidence locations
- Primary publisher/PubMedCentral HTML; VEP result tables
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
18 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Diagram caption Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol. Individual claims | Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods Version: Human Genetics 2025 journal article (online 2024) | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Diagram steps
| Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods Version: Human Genetics 2025 journal article (online 2024) | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Computational evaluation flow Individual claims | Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods Version: Human Genetics 2025 journal article (online 2024) | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Datasets DeepSequence-derived missense-variant DMS datasets, excluding tRNA and viral-family sets; available family sequences and structures support adaptation. Individual claims | Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods Version: Human Genetics 2025 journal article (online 2024) | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits A labeled validation subset selects the structural weighting and model; labels are not used to fit the model parameters. The exact held-out membership after selection remains unextracted. Individual claims | Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods Version: Human Genetics 2025 journal article (online 2024) | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Adaptation Unsupervised family adaptation plus labeled-validation model selection; no fitness-label fitting of model parameters. Individual claims | Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods Version: Human Genetics 2025 journal article (online 2024) | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Metrics Spearman correlation evaluates ranking; AUROC and AUPRC evaluate high/low labels thresholded relative to the wild type. Table 4 compares sequence, structure and combined score variants. Individual claims | Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods Version: Human Genetics 2025 journal article (online 2024) | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Baselines PSSM, EVmutation, DeepSequence, Wavenet and several protein language models; competing scores are obtained from the ESM-1v repository. Individual claims | Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods Version: Human Genetics 2025 journal article (online 2024) | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Leakage controls Family-specific adaptation and validation-label model selection are part of the protocol; it should not be represented as selection without labeled data. Individual claims | Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods Version: Human Genetics 2025 journal article (online 2024) | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Uncertainty One-sided paired Wilcoxon signed-rank comparisons across assay performances are reported for the main method comparison. Individual claims | Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods Version: Human Genetics 2025 journal article (online 2024) | source checked automated source review · 2026-09-16 Audit detailsTask-specific computational methodology and field context checked in the cited primary-source artifact. Source-backed fields, inapplicable evaluator dimensions and unresolved details are distinguished. Numerical results were not reproduced. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
View linked audit checks and correction history
Release 2026-09-29-06401fd5b220 · Record review: needs review
3 source records and release history
- Structure-Informed Protein Language Models are Robust Predictors for Variant Effects · Original source · Human Genetics 2025 journal article (online 2024)
- Structure-informed protein language models are robust predictors for variant effects - PMC · Original source · Human Genetics 2025 journal article (online 2024)
- Structure-Informed Protein Language Models are Robust Predictors for Variant Effects (reviewed HTML snapshot) · Original source · Human Genetics 2025 journal article (online 2024)
Technical metadata and extraction receipts
Stable ID: reported-task-83be0998084c91
- areas
- proteins-complexes
- tasks
- protein variant-effect classification
- entity level
- task
- version
- Not reported
- task
- protein variant-effect classification
- scope note
- Paper-specific evaluation task; protocol completeness requires further extraction.
- benchmark research
- review date: 2026-09-17; status: source_found_structured_extraction_pending; primary sources: evidence-expansion-p2-structure-informed-plm-2025-24c7649edc66; inspected locators: Primary publisher/PubMedCentral HTML; VEP result tables; searched queries: Structure-Informed Protein Language Models are Robust Predictors for Variant Effects primary paper benchmark results; gaps: Artifact returned HTML rather than XML, so XML table parser intentionally rejected it. Prior reviewed observation retained. Complete table extraction and source-version comparison remain pending.; claim scope: Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
- historical missing metadata
- protocol version: not_reported_in_legacy_extract
- metadata review scope
- historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
- legacy kinds
- benchmark
- entity classification
- review date: 2026-09-17; rationale: This source-scoped record identifies the biological prediction task and holds its paper context. Preserve the existing task identity; exact split, model adaptation and scoring remain in linked evaluations or separate protocol records.; source ids: evidence-benchmark-structure-informed-html; source locator: Results: Structure-Informed pLMs Predict Variant Fitness Robustly; Tables 2 and 4; Methods: Datasets, Mutation effect datasets, Competing methods; ambiguities: A paper- or suite-specific task may constrain some inputs or metrics; that alone does not make it interchangeable with a complete versioned protocol. No protocol equivalence is inferred.; Some legacy profile Entity type facts use the generic phrase computational evaluation protocol. That boilerplate is not sufficient to establish a single fixed protocol identity or to merge this task with another protocol record.