Datasets
Experimental DMS variant-effect measurements across activity, binding, expression, organismal fitness and stability categories.
Task
protein variant fitness prediction
ProteinGym substitution assays evaluate agreement between a fixed protein-model scoring method and measured variant effects.
Sources
Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I captionOverview
Metrics
Spearman rank correlation compares measured variant effects with model scores; Table I groups substitution assays by functional category.
Allowed inputs
Variant protein sequences for model scoring.
Sources
Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I captionEvaluation procedure diagram
How it worksComputational evaluation flow
Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.
Sources
Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I captionlimited source coverage · Automated source review, 2026-09-16. All specifications and missing details
Results
Results are available, but no reviewed comparison panel is linked in this release.
All evaluations
1 evaluation · 1 result. Different protocols are not a single leaderboard.
Filter evaluations
Applied filters: All linked evaluations
| Tested configuration | Protocol and dataset | Finding | Evidence and details |
|---|---|---|---|
| Pipeline: ESM2 OFS pseudo-perplexity | Task: protein variant fitness prediction Dataset: ProteinGym substitutions | 0.403 Spearman rho unitless · unknown Uncertainty: Not reported Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceESM2 OFS pseudo-perplexity: protein variant fitness prediction authors’ zero-shot ESM2 OFS pseudo-perplexity evaluation; aggregate mean across ProteinGym substitution assays Aggregation: Not reported Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · Published PDF page 6 (033014-6), Table I, ESM2: OFS PP row, Aggregate mean column |
Source checking is not independent reproduction. Release 2026-09-29-06401fd5b220.
Methods and evaluation design
Procedure, tasks and evaluated configurations
How it works
Evaluation methodology
Experimental DMS variant-effect measurements across activity, binding, expression, organismal fitness and stability categories. The downstream assay evaluation is described as zero-shot; a separate approximation-model training partition must not be treated as the assay test split. Spearman rank correlation compares measured variant effects with model scores; Table I groups substitution assays by functional category. CARP, ESM-1v, Progen2 M, RITA L and Tranception L without retrieval use baseline data sourced from ProteinGym; ESM2 scoring variants are also compared. The cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim.
Sources
Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I captionRecorded evaluations
Each evaluation records what was tested and under which conditions.
Run instructions
No runnable recipe has been reviewed for this task. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.
A task describes a biological question. Choose a linked protocol to obtain concrete split and scoring instructions.
Strengths, limitations and unresolved questions
Strengths and limitations
Strengths and considerations
No source-reviewed explanatory claims are recorded here yet.
Limitations and conditions
- This record concerns substitution assays, separate from the indel and other demonstrations. The OFS-training identity filter does not certify that the ESM-2 pretraining corpus is independent of the benchmark.
Sources
Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption
Profile review details
Relevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged.
Stable record: reported-task-c7a8a372f77886Specifications
Inputs, training, access and other details
Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.
| Property | Description and evidence |
|---|---|
| Datasets | Experimental DMS variant-effect measurements across activity, binding, expression, organismal fitness and stability categories.SourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption |
| Splits | The downstream assay evaluation is described as zero-shot; a separate approximation-model training partition must not be treated as the assay test split.SourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption |
| Metrics | Spearman rank correlation compares measured variant effects with model scores; Table I groups substitution assays by functional category.SourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption |
| Baselines | CARP, ESM-1v, Progen2 M, RITA L and Tranception L without retrieval use baseline data sourced from ProteinGym; ESM2 scoring variants are also compared.SourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption |
| Leakage controls | Training of the OFS approximation model withholds sequences sharing more than 50% identity with ProteinGym reference sequences. This is an explicit control on the approximation-training corpus; it is not a claim that the original ESM-2 pretraining corpus excludes those proteins.SourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · PRX Life paper, Methods: OFS model training; extracted PDF lines 918–934 |
| Uncertainty | The cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim. · Not reported in inspected sourcesSourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption |
| Entity type | Paper-specific computational evaluation protocol.SourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption |
| Organisms | ProteinGym assay-specific taxa.SourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption |
| Assays | DMS substitution measurements across functional categories.SourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption |
| Allowed inputs | Variant protein sequences for model scoring.SourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption |
| Adaptation | Zero-shot downstream assay evaluation; OFS approximation-model fitting is separate.SourcesPseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption |
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Papers and result coverage
Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
| Paper or primary resource | Version | Reference |
|---|---|---|
| Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation | PRX Life 2025 journal article | Read source |
Historical gaps recorded on 2026-09-17
The catalogue now holds 1 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.
- Pinned source PDF retained; source-specific assay aggregation and substitution/indel scope require full table mapping. Existing result preserved; structured extraction pending.
Search and extraction details
source found structured extraction pending
Searches
- Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation primary paper benchmark results
Evidence locations
- Primary APS paperPDF; fitness-estimation comparison tables
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
17 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Diagram caption Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol. Individual claims | Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption Version: PRX Life 2025 journal article | source checked automated source review · 2026-09-16 Audit detailsRelevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Diagram steps
| Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption Version: PRX Life 2025 journal article | source checked automated source review · 2026-09-16 Audit detailsRelevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Diagram title Computational evaluation flow Individual claims | Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption Version: PRX Life 2025 journal article | source checked automated source review · 2026-09-16 Audit detailsRelevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Datasets Experimental DMS variant-effect measurements across activity, binding, expression, organismal fitness and stability categories. Individual claims | Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption Version: PRX Life 2025 journal article | source checked automated source review · 2026-09-16 Audit detailsRelevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Splits The downstream assay evaluation is described as zero-shot; a separate approximation-model training partition must not be treated as the assay test split. Individual claims | Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption Version: PRX Life 2025 journal article | source checked automated source review · 2026-09-16 Audit detailsRelevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Adaptation Zero-shot downstream assay evaluation; OFS approximation-model fitting is separate. Individual claims | Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption Version: PRX Life 2025 journal article | source checked automated source review · 2026-09-16 Audit detailsRelevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Metrics Spearman rank correlation compares measured variant effects with model scores; Table I groups substitution assays by functional category. Individual claims | Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption Version: PRX Life 2025 journal article | source checked automated source review · 2026-09-16 Audit detailsRelevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Baselines CARP, ESM-1v, Progen2 M, RITA L and Tranception L without retrieval use baseline data sourced from ProteinGym; ESM2 scoring variants are also compared. Individual claims | Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption Version: PRX Life 2025 journal article | source checked automated source review · 2026-09-16 Audit detailsRelevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Leakage controls Training of the OFS approximation model withholds sequences sharing more than 50% identity with ProteinGym reference sequences. This is an explicit control on the approximation-training corpus; it is not a claim that the original ESM-2 pretraining corpus excludes those proteins. Individual claims | Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation PRX Life paper, Methods: OFS model training; extracted PDF lines 918–934 Version: PRX Life 2025 journal article | source checked automated source review · 2026-09-16 Audit detailsRelevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
| Uncertainty The cited text-accessible evaluation sections give no confidence-interval, resampling or repeat-run error-bar specification. Image-only tables and uninspected supplements are outside this absence claim. Individual claims | Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption Version: PRX Life 2025 journal article | unreported automated source review · 2026-09-16 Audit detailsRelevant full-paper computational evaluation sections, tables/captions and cited supplementary task passages were reviewed. Reporting omissions are scoped to the inspected sources. Original numerical results are unchanged. Field: Source artifact SHA-256: Hash scope: Hash scope not separately documented; inspect source record |
Sources and history
View linked audit checks and correction history
Release 2026-09-29-06401fd5b220 · Record review: needs review
2 source records and release history
- Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · Original source · PRX Life 2025 journal article
- Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation · Original source · PRX Life 2025 journal article
Technical metadata and extraction receipts
Stable ID: reported-task-c7a8a372f77886
- areas
- proteins-complexes
- tasks
- protein variant fitness prediction
- entity level
- task
- version
- Not reported
- task
- protein variant fitness prediction
- scope note
- Paper-specific evaluation task; protocol completeness requires further extraction.
- benchmark research
- review date: 2026-09-17; status: source_found_structured_extraction_pending; primary sources: evidence-expansion-p2-esm2-ofs-fitness-2025-085ef646f11b; inspected locators: Primary APS paperPDF; fitness-estimation comparison tables; searched queries: Pseudo-perplexity in One Fell Swoop for Protein Fitness Estimation primary paper benchmark results; gaps: Pinned source PDF retained; source-specific assay aggregation and substitution/indel scope require full table mapping. Existing result preserved; structured extraction pending.; claim scope: Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.
- historical missing metadata
- protocol version: not_reported_in_legacy_extract
- metadata review scope
- historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
- legacy kinds
- benchmark
- entity classification
- review date: 2026-09-17; rationale: This source-scoped record identifies the biological prediction task and holds its paper context. Preserve the existing task identity; exact split, model adaptation and scoring remain in linked evaluations or separate protocol records.; source ids: esm2-ofs-fitness-2025; source locator: ProteinGym evaluation section and Table I; PDF text lines 346–369 and Table I caption; ambiguities: A paper- or suite-specific task may constrain some inputs or metrics; that alone does not make it interchangeable with a complete versioned protocol. No protocol equivalence is inferred.; Some legacy profile Entity type facts use the generic phrase computational evaluation protocol. That boilerplate is not sufficient to establish a single fixed protocol identity or to merge this task with another protocol record.