rewirebio.iobenchmarks
Task

vaccine-antigen candidate prediction

Vaccine-antigen classification tests transfer by withholding one organism’s labeled protein examples.

SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

4 evaluations · 22 results

Overview

Datasets

Previously curated vaccine-antigen positive/negative datasets used by Vaxign-ML and Vaxign-DL.

Metrics

Accuracy, sensitivity, specificity, weighted F1, MCC, AUROC and AUPRC.

Allowed inputs

Protein sequence representations.

SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39
Evaluation procedure diagram
How it worksComputational evaluation flow
Computational evaluation flow1. Input: Protein sequence representations.. Then: 2. Evaluation: Leave-one-pathogen-out evaluation trains on the remaining organisms and predicts the withheld organism’s labels.. Then: 3. Readout: Accuracy, sensitivity, specificity, weighted F1, MCC, AUROC and AUPRC.Computational evaluation flow1. Input: Protein sequence representations.. Then: 2. Evaluation: Leave-one-pathogen-out evaluation trains on the remaining organisms and predicts the withheld organism’s labels.. Then: 3. Readout: Accuracy, sensitivity, specificity, weighted F1, MCC, AUROC and AUPRC.Computational evaluation flow1. Input: Protein sequence representations.. Then: 2. Evaluation: Leave-one-pathogen-out evaluation trains on the remaining organisms and predicts the withheld organism’s labels.. Then: 3. Readout: Accuracy, sensitivity, specificity, weighted F1, MCC, AUROC and AUPRC.

Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.

SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

limited source coverage · Automated source review, 2026-09-16. All specifications and missing details

4 recorded evaluations, 22 metric rows. Explore 7 source-scoped charts.

View coverage and remaining gaps across all benchmarks

Results

Each comparison retains its reviewed evaluation scope, dataset and metric. Results are shown without a pooled ranking.

Protective-antigen classification feature comparison · Table 1.

accuracy (fraction) · Higher values are better.

Protective-antigen classification feature comparison (vaccine-antigen candidate prediction) · Protective-antigen classification feature comparison

Evidence origin: Author-reported evaluation.

Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Table 1.: Val Accuracy, Protective-antigen classification feature comparison
  • Reported± values preserved; uncertainty type and resampling denominator not established in inspected methods.
  • Published prose states specificity equal but table contains differences; table values retained.
Comparison details and limitations

Original 509 features vs ESM 1,280 features vs combined 1,789 features. 372 protection-positive and 3,720 protection-negative samples.

  • No interval assigned unless printed in source cell.

Automated source review: 2026-09-17. Numerical source review does not establish independent reproduction.

Dots show point estimates. Whiskers show only explicitly defined uncertainty (standard deviation, standard error or a labelled interval); their definitions remain in Table. Unresolved uncertainty is not plotted. Differences do not establish statistical significance.

Showing 3 of 3 matching rows.

Tested configuration
00.250.50.751
Reported score
  1. Combined0.97 ± 0.004
  2. Original0.96 ± 0.008
  3. ESM Only0.96 ± 0.005

Methods and evaluation design

Procedure, tasks and evaluated configurations

How it works

Evaluation methodology

Previously curated vaccine-antigen positive/negative datasets used by Vaxign-ML and Vaxign-DL. Leave-one-pathogen-out evaluation trains on the remaining organisms and predicts the withheld organism’s labels. Accuracy, sensitivity, specificity, weighted F1, MCC, AUROC and AUPRC. Vaxign-ML and Vaxign-DL are the referenced baseline datasets/methods. Dataset curation removes positive-protein homologs above 30% sequence similarity and selects negative proteins below 30% similarity to positives. The LOPOV experiment additionally withholds one of ten pathogens. These controls do not establish that sequences were absent from ESM-1b pretraining.

SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39; Methods: Collection of Positive and Negative protein sequences; Leave-one-pathogen-out validation

Evaluation design

Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.

These source-backed links do not make different protocols or scores interchangeable.

Recorded evaluations

Each evaluation records what was tested and under which conditions.

Run this benchmark

Choose a concrete protocol before running an evaluation. Its inputs, split and scoring rules determine which results can be compared.

Run instructions

No runnable recipe has been reviewed for this task. Dataset access, model requirements, licences and compute requirements must be checked against its sources before execution.

A task describes a biological question. Choose a linked protocol to obtain concrete split and scoring instructions.

Strengths, limitations and unresolved questions

Strengths and limitations

Strengths and considerations

No source-reviewed explanatory claims are recorded here yet.

Profile review details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Stable record: reported-task-47465954d606e6

Specifications

Inputs, training, access and other details

Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
PropertyDescription and evidence
DatasetsPreviously curated vaccine-antigen positive/negative datasets used by Vaxign-ML and Vaxign-DL.
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39
SplitsLeave-one-pathogen-out evaluation trains on the remaining organisms and predicts the withheld organism’s labels.
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39
MetricsAccuracy, sensitivity, specificity, weighted F1, MCC, AUROC and AUPRC.
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39
BaselinesVaxign-ML and Vaxign-DL are the referenced baseline datasets/methods.
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39
Leakage controlsDataset curation removes positive-protein homologs above 30% sequence similarity and selects negative proteins below 30% similarity to positives. The LOPOV experiment additionally withholds one of ten pathogens. These controls do not establish that sequences were absent from ESM-1b pretraining.
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: Collection of Positive and Negative protein sequences; Leave-one-pathogen-out validation
UncertaintyTables 1–3 include ± terms for most metrics but not AUROC. Their captions and the Deep learning pipeline section do not define whether those terms are standard deviations, standard errors or confidence intervals, or specify their repetition count. · Not reported in inspected sources
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: Deep learning pipeline; Tables 1–3
Entity typePaper-specific computational evaluation protocol.
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39
OrganismsMultiple pathogen species in the curated antigen evaluation.
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39
AssaysVaccine-antigen positive/negative annotations.
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39
Allowed inputsProtein sequence representations.
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39
AdaptationSupervised candidate classification with leave-one-pathogen-out evaluation.
SourcesEnhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features · Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Papers and result coverage

Last literature check: 2026-09-17. Primary-source discovery and table/protocol screening; source checked is not independently reproduced. Raw acquisitions not automatically numerical publication approval.

Paper or primary resourceVersionReference
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Featurespreprint version in PMCRead source
DOI: 10.1101/2024.09.04.611295
Historical gaps recorded on 2026-09-17

The catalogue now holds 22 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.

  • Independent batch review before import; preserve existing observation identities.
Search and extraction details

complete comparison tables extracted pending publication review

Searches

  • Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features primary paper benchmark results

Evidence locations

  • Tables1–2; Performance analysis and feature sets

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

17 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-21-adc15377fab2
Property and statementOriginal source and locationReview and provenance
Diagram caption
Conceptual summary of the cited evaluation; exact task configuration and source version remain part of the protocol.
Individual claims
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features

Original source ↗

Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

Version: preprint version in PMC
Retrieved: 2026-09-16T10:41:06Z

source checked

automated source review · 2026-09-16

Audit details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Field: attributes.profile.diagram.caption

Source artifact SHA-256: b76fff917addd0e9ff8a2fc843496132ecf832d3ceef248e3edf2dbc78baab5f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Diagram steps
  • Input: Protein sequence representations.
  • Evaluation: Leave-one-pathogen-out evaluation trains on the remaining organisms and predicts the withheld organism’s labels.
  • Readout: Accuracy, sensitivity, specificity, weighted F1, MCC, AUROC and AUPRC.
Individual claims
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features

Original source ↗

Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

Version: preprint version in PMC
Retrieved: 2026-09-16T10:41:06Z

source checked

automated source review · 2026-09-16

Audit details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Field: attributes.profile.diagram.steps

Source artifact SHA-256: b76fff917addd0e9ff8a2fc843496132ecf832d3ceef248e3edf2dbc78baab5f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Diagram title
Computational evaluation flow
Individual claims
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features

Original source ↗

Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

Version: preprint version in PMC
Retrieved: 2026-09-16T10:41:06Z

source checked

automated source review · 2026-09-16

Audit details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Field: attributes.profile.diagram.title

Source artifact SHA-256: b76fff917addd0e9ff8a2fc843496132ecf832d3ceef248e3edf2dbc78baab5f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Datasets
Previously curated vaccine-antigen positive/negative datasets used by Vaxign-ML and Vaxign-DL.
Individual claims
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features

Original source ↗

Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

Version: preprint version in PMC
Retrieved: 2026-09-16T10:41:06Z

source checked

automated source review · 2026-09-16

Audit details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Field: attributes.profile.facts.0.value

Source artifact SHA-256: b76fff917addd0e9ff8a2fc843496132ecf832d3ceef248e3edf2dbc78baab5f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Splits
Leave-one-pathogen-out evaluation trains on the remaining organisms and predicts the withheld organism’s labels.
Individual claims
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features

Original source ↗

Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

Version: preprint version in PMC
Retrieved: 2026-09-16T10:41:06Z

source checked

automated source review · 2026-09-16

Audit details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Field: attributes.profile.facts.1.value

Source artifact SHA-256: b76fff917addd0e9ff8a2fc843496132ecf832d3ceef248e3edf2dbc78baab5f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Adaptation
Supervised candidate classification with leave-one-pathogen-out evaluation.
Individual claims
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features

Original source ↗

Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

Version: preprint version in PMC
Retrieved: 2026-09-16T10:41:06Z

source checked

automated source review · 2026-09-16

Audit details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Field: attributes.profile.facts.10.value

Source artifact SHA-256: b76fff917addd0e9ff8a2fc843496132ecf832d3ceef248e3edf2dbc78baab5f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Metrics
Accuracy, sensitivity, specificity, weighted F1, MCC, AUROC and AUPRC.
Individual claims
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features

Original source ↗

Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

Version: preprint version in PMC
Retrieved: 2026-09-16T10:41:06Z

source checked

automated source review · 2026-09-16

Audit details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Field: attributes.profile.facts.2.value

Source artifact SHA-256: b76fff917addd0e9ff8a2fc843496132ecf832d3ceef248e3edf2dbc78baab5f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Baselines
Vaxign-ML and Vaxign-DL are the referenced baseline datasets/methods.
Individual claims
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features

Original source ↗

Methods: leave-one-pathogen-out validation; Performance evaluation; Results: long-protein coverage; cached text lines 18, 25–26, 39

Version: preprint version in PMC
Retrieved: 2026-09-16T10:41:06Z

source checked

automated source review · 2026-09-16

Audit details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Field: attributes.profile.facts.3.value

Source artifact SHA-256: b76fff917addd0e9ff8a2fc843496132ecf832d3ceef248e3edf2dbc78baab5f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Leakage controls
Dataset curation removes positive-protein homologs above 30% sequence similarity and selects negative proteins below 30% similarity to positives. The LOPOV experiment additionally withholds one of ten pathogens. These controls do not establish that sequences were absent from ESM-1b pretraining.
Individual claims
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features

Original source ↗

Methods: Collection of Positive and Negative protein sequences; Leave-one-pathogen-out validation

Version: preprint version in PMC
Retrieved: 2026-09-16T10:41:06Z

source checked

automated source review · 2026-09-16

Audit details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Field: attributes.profile.facts.4.value

Source artifact SHA-256: b76fff917addd0e9ff8a2fc843496132ecf832d3ceef248e3edf2dbc78baab5f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Uncertainty
Tables 1–3 include ± terms for most metrics but not AUROC. Their captions and the Deep learning pipeline section do not define whether those terms are standard deviations, standard errors or confidence intervals, or specify their repetition count.
Individual claims
Enhancing Vaxign-DL for Vaccine Candidate Prediction with added ESM-Generated Features

Original source ↗

Methods: Deep learning pipeline; Tables 1–3

Version: preprint version in PMC
Retrieved: 2026-09-16T10:41:06Z

unreported

automated source review · 2026-09-16

Audit details

Targeted full-paper and supplement review of the outstanding task fields, with original dataset metadata checked where accessible. Source-scoped omissions are explicit; no independent benchmark reproduction or numerical-result change.

Field: attributes.profile.facts.5.value

Source artifact SHA-256: b76fff917addd0e9ff8a2fc843496132ecf832d3ceef248e3edf2dbc78baab5f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-21-adc15377fab2 · Record review: needs review

2 source records and release historyDownload this release (gzip)
Technical metadata and record identifiers

Technical metadata for this record is tracked under stable ID reported-task-47465954d606e6.

Related records

Suggest a correction