rewirebio.iobenchmarks
Evidence claim

Relevance of Mega-scale dataset stability ranking (Chu et al. 2024) to "Assess methods for protein stability experiments"

Use case
Assess methods for protein stability experiments
Assessed by
Mega-scale dataset stability ranking, Spearman correlation (Chu et al. 2024 Table 1)
Relevance
Direct evidence
Endpoint
Spearman correlation for Rosetta, MUPro, RaSP, ELASPIC-2 and two ESM-2 checkpoints on Mega-scale dataset

Held-out protein domains from the cDNA display proteolysis set directly measure how well each method ranks substitutions by folding stability.

6 reviewed evaluations

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

21 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-11-21-adc15377fab2
Property and statementOriginal source and locationReview and provenance
Citation locations
1 values
  • source id
    protein-stability-20261009-source-chu2024
    locator
    Table 1 row 'Mega-scale dataset'; Table 2; Results
Context-only references
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset

Original source ↗

protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results

Version: PLOS Computational Biology 20(7):e1012248, published 2024-07-22; PMC11293664 full-text XML
Retrieved: 2026-10-09T21:01:08Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.citation_locators

Source artifact SHA-256: bb9830cbd6bab1b3e0ddbb52edb2a9afd4b06e427dbfeb19e6a07f843ebd914f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Comparison group
chu2024-transfer
Context-only references
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset

Original source ↗

protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results

Version: PLOS Computational Biology 20(7):e1012248, published 2024-07-22; PMC11293664 full-text XML
Retrieved: 2026-10-09T21:01:08Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.comparison_group

Source artifact SHA-256: bb9830cbd6bab1b3e0ddbb52edb2a9afd4b06e427dbfeb19e6a07f843ebd914f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Comparison title
Transfer from mega-scale domains to larger proteins (Chu et al. 2024)
Context-only references
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset

Original source ↗

protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results

Version: PLOS Computational Biology 20(7):e1012248, published 2024-07-22; PMC11293664 full-text XML
Retrieved: 2026-10-09T21:01:08Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.comparison_title

Source artifact SHA-256: bb9830cbd6bab1b3e0ddbb52edb2a9afd4b06e427dbfeb19e6a07f843ebd914f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Constraints
2 values
  • Inspect every linked evaluation's source locator before citing a result.
  • Do not combine this mapping's evaluations with any other protocol's results; datasets, splits, structures and predictor versions differ between sources.
Context-only references
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset

Original source ↗

protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results

Version: PLOS Computational Biology 20(7):e1012248, published 2024-07-22; PMC11293664 full-text XML
Retrieved: 2026-10-09T21:01:08Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.constraints

Source artifact SHA-256: bb9830cbd6bab1b3e0ddbb52edb2a9afd4b06e427dbfeb19e6a07f843ebd914f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Endpoint
Spearman correlation for Rosetta, MUPro, RaSP, ELASPIC-2 and two ESM-2 checkpoints on Mega-scale dataset
Context-only references
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset

Original source ↗

protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results

Version: PLOS Computational Biology 20(7):e1012248, published 2024-07-22; PMC11293664 full-text XML
Retrieved: 2026-10-09T21:01:08Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.endpoint

Source artifact SHA-256: bb9830cbd6bab1b3e0ddbb52edb2a9afd4b06e427dbfeb19e6a07f843ebd914f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Excluded evaluations
1 values
  • id
    protein-stability-20261009-eval-chu2024-esm-therm-mega-scale
    reason
    Table 1 caption: every other method is scored on point mutations only, while ESM therm on the mega-scale set is also scored on multi-point variants, so its row is not matched. Excluded from this comparison at independent review 2026-10-09; its result stays recorded.
Context-only references
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset

Original source ↗

protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results

Version: PLOS Computational Biology 20(7):e1012248, published 2024-07-22; PMC11293664 full-text XML
Retrieved: 2026-10-09T21:01:08Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.excluded_evaluations

Source artifact SHA-256: bb9830cbd6bab1b3e0ddbb52edb2a9afd4b06e427dbfeb19e6a07f843ebd914f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Claimed field
links:assessed_by:protein-stability-20261009-protocol-chu2024-mega-scale
Context-only references
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset

Original source ↗

protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results

Version: PLOS Computational Biology 20(7):e1012248, published 2024-07-22; PMC11293664 full-text XML
Retrieved: 2026-10-09T21:01:08Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.field

Source artifact SHA-256: bb9830cbd6bab1b3e0ddbb52edb2a9afd4b06e427dbfeb19e6a07f843ebd914f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Headline metric
spearman-correlation
Context-only references
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset

Original source ↗

protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results

Version: PLOS Computational Biology 20(7):e1012248, published 2024-07-22; PMC11293664 full-text XML
Retrieved: 2026-10-09T21:01:08Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.headline_metric

Source artifact SHA-256: bb9830cbd6bab1b3e0ddbb52edb2a9afd4b06e427dbfeb19e6a07f843ebd914f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Limitations
6 values
  • ESM therm was fine-tuned by the authors of this paper; the other methods were run by them as comparators.
  • Only Spearman correlation is printed; no RMSE, calibration or uncertainty.
  • The larger-protein benchmark was limited to six ProteinGym datasets and BglB because of Rosetta's compute cost (Methods P25).
  • ESM therm is excluded from this comparison: it was scored on point and multi-point variants, the other methods on point mutations only (Table 1 caption).
  • Metrics are aggregated over test-set-only domains of 40-72 residues, the same assay the model was trained on.
  • The paper does not say who produced the comparator scores on the mega-scale test-set-only domains.
Context-only references
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset

Original source ↗

protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results

Version: PLOS Computational Biology 20(7):e1012248, published 2024-07-22; PMC11293664 full-text XML
Retrieved: 2026-10-09T21:01:08Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.limitations

Source artifact SHA-256: bb9830cbd6bab1b3e0ddbb52edb2a9afd4b06e427dbfeb19e6a07f843ebd914f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

Pinned records
15 values
  • record id
    protein-stability-20261009-eval-chu2024-elaspic-2-mega-scale
    fields
    • status
    • links
    • attributes.origin
    • attributes.comparison
    sha256
    e25c5cbcf061bdf37a2889e7c7c57241c73ec7fb45c720abc70efd0dee392016
  • record id
    protein-stability-20261009-eval-chu2024-esm-2-35m-mega-scale
    fields
    • status
    • links
    • attributes.origin
    • attributes.comparison
    sha256
    772c3670218407282e9243eda52d83c288add1f7ec1057e8b0c89fa94e9fa865
  • record id
    protein-stability-20261009-eval-chu2024-esm-2-3b-mega-scale
    fields
    • status
    • links
    • attributes.origin
    • attributes.comparison
    sha256
    7703a5454b4dd581fb085c92c54c90a71b62e8f4cd8661f8113292b572351db7
  • record id
    protein-stability-20261009-eval-chu2024-mupro-mega-scale
    fields
    • status
    • links
    • attributes.origin
    • attributes.comparison
    sha256
    63fb394b916db56e3cc8f2e567676b0373c055f2026d5e346a33f8faa51c293f
  • record id
    protein-stability-20261009-eval-chu2024-rasp-mega-scale
    fields
    • status
    • links
    • attributes.origin
    • attributes.comparison
    sha256
    f2d117ca5287f983651434e41eaef003d1ded77a6dd0e9538a724a45108ce860
  • record id
    protein-stability-20261009-eval-chu2024-rosetta-mega-scale
    fields
    • status
    • links
    • attributes.origin
    • attributes.comparison
    sha256
    d3fa3b024ea49abc67c2dc09b4de5e40328e880164275d8458f8d1ce76a23009
  • record id
    protein-stability-20261009-protocol-chu2024-mega-scale
    fields
    • status
    • name
    • description
    • facets
    • source_ids
    • links
    • attributes
    sha256
    c804d78c9b5f79cf44f4d8a64676edb8f2283c505026cabbbc27550c8c10b6f0
  • record id
    protein-stability-20261009-result-chu2024-elaspic-2-mega-scale-spearman-correlation
    fields
    • status
    • attributes.metric
    • attributes.metric_qualifier
    • attributes.printed_value
    • attributes.numeric_value
    sha256
    a8ca3493739c96fe78d2bbe8de27e56d2be0acceaecfaa608ec132732189a8d0
  • record id
    protein-stability-20261009-result-chu2024-esm-2-35m-mega-scale-spearman-correlation
    fields
    • status
    • attributes.metric
    • attributes.metric_qualifier
    • attributes.printed_value
    • attributes.numeric_value
    sha256
    377ec36ac90c4520ddf018495ac0320ba5dd4f64aa1507a0ccec9230c688f6ab
  • record id
    protein-stability-20261009-result-chu2024-esm-2-3b-mega-scale-spearman-correlation
    fields
    • status
    • attributes.metric
    • attributes.metric_qualifier
    • attributes.printed_value
    • attributes.numeric_value
    sha256
    1d8638baa92c85d45dcec1e409d6410265c8e7cf083755798a972ecc420cb20e
  • record id
    protein-stability-20261009-result-chu2024-mupro-mega-scale-spearman-correlation
    fields
    • status
    • attributes.metric
    • attributes.metric_qualifier
    • attributes.printed_value
    • attributes.numeric_value
    sha256
    7ee99a22eed951928dfb6e29fe3976f6fe4ff9f7306a5a00b6cd39d94628ac68
  • record id
    protein-stability-20261009-result-chu2024-rasp-mega-scale-spearman-correlation
    fields
    • status
    • attributes.metric
    • attributes.metric_qualifier
    • attributes.printed_value
    • attributes.numeric_value
    sha256
    a8ca3493739c96fe78d2bbe8de27e56d2be0acceaecfaa608ec132732189a8d0
  • record id
    protein-stability-20261009-result-chu2024-rosetta-mega-scale-spearman-correlation
    fields
    • status
    • attributes.metric
    • attributes.metric_qualifier
    • attributes.printed_value
    • attributes.numeric_value
    sha256
    f6bbc25f6760c82aba00ad40db1869002ccd85a00ef60fc81e150df4f383ca41
  • record id
    protein-stability-20261009-source-chu2024
    fields
    • status
    • attributes.artifact_sha256
    sha256
    e959594a702e1cfb9147c235f9a31116ada6106293e06b6e23185a14a95803b0
  • record id
    use-case-protein-stability
    fields
    • status
    • name
    • description
    • facets
    • attributes.question
    • attributes.decision
    • attributes.inputs
    • attributes.output
    • attributes.exclusions
    • attributes.setting
    • attributes.clinical_scope
    • attributes.intended_users
    sha256
    2dd1d0e139a17e51ae72cc5945196ffbb366fe4daa4c20c4efd950346a30a54f
Context-only references
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset

Original source ↗

protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results

Version: PLOS Computational Biology 20(7):e1012248, published 2024-07-22; PMC11293664 full-text XML
Retrieved: 2026-10-09T21:01:08Z

not individually reviewed

No individual claim review recorded

Audit details

Field: attributes.pins

Source artifact SHA-256: bb9830cbd6bab1b3e0ddbb52edb2a9afd4b06e427dbfeb19e6a07f843ebd914f

Hash scope: Hash scope not separately documented; inspect source record

Inspected artifact

How to read the evidence

Sources and history

Release 2026-11-21-adc15377fab2 · Record review: source checked

1 source record and release historyDownload this release (gzip)
Technical metadata and extraction receipts

Stable ID: use-case-mapping-protein-stability-20261009-chu2024-mega-scale

field
links:assessed_by:protein-stability-20261009-protocol-chu2024-mega-scale
value
protein-stability-20261009-protocol-chu2024-mega-scale
relevance
direct
endpoint
Spearman correlation for Rosetta, MUPro, RaSP, ELASPIC-2 and two ESM-2 checkpoints on Mega-scale dataset
rationale
Held-out protein domains from the cDNA display proteolysis set directly measure how well each method ranks substitutions by folding stability.
constraints
Inspect every linked evaluation's source locator before citing a result.; Do not combine this mapping's evaluations with any other protocol's results; datasets, splits, structures and predictor versions differ between sources.
limitations
ESM therm was fine-tuned by the authors of this paper; the other methods were run by them as comparators.; Only Spearman correlation is printed; no RMSE, calibration or uncertainty.; The larger-protein benchmark was limited to six ProteinGym datasets and BglB because of Rosetta's compute cost (Methods P25).; ESM therm is excluded from this comparison: it was scored on point and multi-point variants, the other methods on point mutations only (Table 1 caption).; Metrics are aggregated over test-set-only domains of 40-72 residues, the same assay the model was trained on.; The paper does not say who produced the comparator scores on the mega-scale test-set-only domains.
revision
1
reason
Add primary-source stability predictor comparison evidence from the protein stability use-case pass (2026-10-09).
citation locators
source id: protein-stability-20261009-source-chu2024; locator: Table 1 row 'Mega-scale dataset'; Table 2; Results
source locator
protein-stability-20261009-source-chu2024: Table 1 row 'Mega-scale dataset'; Table 2; Results
comparison group
chu2024-transfer
comparison title
Transfer from mega-scale domains to larger proteins (Chu et al. 2024)
headline metric
spearman-correlation
stratum label
Mega-scale held-out domains
stratum order
1
excluded evaluations
id: protein-stability-20261009-eval-chu2024-esm-therm-mega-scale; reason: Table 1 caption: every other method is scored on point mutations only, while ESM therm on the mega-scale set is also scored on multi-point variants, so its row is not matched. Excluded from this comparison at independent review 2026-10-09; its result stays recorded.
reviewed evaluations
protein-stability-20261009-eval-chu2024-elaspic-2-mega-scale; protein-stability-20261009-eval-chu2024-esm-2-35m-mega-scale; protein-stability-20261009-eval-chu2024-esm-2-3b-mega-scale; protein-stability-20261009-eval-chu2024-mupro-mega-scale; protein-stability-20261009-eval-chu2024-rasp-mega-scale; protein-stability-20261009-eval-chu2024-rosetta-mega-scale
review
method: automated-source-review; reviewer: claude; reviewer note: Separate Claude review agent, independent of the extractor; no human review claimed; reviewed at: 2026-10-09T23:59:00Z; note: Independent review of endpoint fit, relevance grade, limitations, exclusions and grouping against the use case and the source; every value of the listed evaluations was checked against the source table. No execution, reproduction, human scientific review or clinical validation.
pins
record id: protein-stability-20261009-eval-chu2024-elaspic-2-mega-scale; fields: status; links; attributes.origin; attributes.comparison; sha256: e25c5cbcf061bdf37a2889e7c7c57241c73ec7fb45c720abc70efd0dee392016; record id: protein-stability-20261009-eval-chu2024-esm-2-35m-mega-scale; fields: status; links; attributes.origin; attributes.comparison; sha256: 772c3670218407282e9243eda52d83c288add1f7ec1057e8b0c89fa94e9fa865; record id: protein-stability-20261009-eval-chu2024-esm-2-3b-mega-scale; fields: status; links; attributes.origin; attributes.comparison; sha256: 7703a5454b4dd581fb085c92c54c90a71b62e8f4cd8661f8113292b572351db7; record id: protein-stability-20261009-eval-chu2024-mupro-mega-scale; fields: status; links; attributes.origin; attributes.comparison; sha256: 63fb394b916db56e3cc8f2e567676b0373c055f2026d5e346a33f8faa51c293f; record id: protein-stability-20261009-eval-chu2024-rasp-mega-scale; fields: status; links; attributes.origin; attributes.comparison; sha256: f2d117ca5287f983651434e41eaef003d1ded77a6dd0e9538a724a45108ce860; record id: protein-stability-20261009-eval-chu2024-rosetta-mega-scale; fields: status; links; attributes.origin; attributes.comparison; sha256: d3fa3b024ea49abc67c2dc09b4de5e40328e880164275d8458f8d1ce76a23009; record id: protein-stability-20261009-protocol-chu2024-mega-scale; fields: status; name; description; facets; source_ids; links; attributes; sha256: c804d78c9b5f79cf44f4d8a64676edb8f2283c505026cabbbc27550c8c10b6f0; record id: protein-stability-20261009-result-chu2024-elaspic-2-mega-scale-spearman-correlation; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: a8ca3493739c96fe78d2bbe8de27e56d2be0acceaecfaa608ec132732189a8d0; record id: protein-stability-20261009-result-chu2024-esm-2-35m-mega-scale-spearman-correlation; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 377ec36ac90c4520ddf018495ac0320ba5dd4f64aa1507a0ccec9230c688f6ab; record id: protein-stability-20261009-result-chu2024-esm-2-3b-mega-scale-spearman-correlation; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 1d8638baa92c85d45dcec1e409d6410265c8e7cf083755798a972ecc420cb20e; record id: protein-stability-20261009-result-chu2024-mupro-mega-scale-spearman-correlation; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: 7ee99a22eed951928dfb6e29fe3976f6fe4ff9f7306a5a00b6cd39d94628ac68; record id: protein-stability-20261009-result-chu2024-rasp-mega-scale-spearman-correlation; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: a8ca3493739c96fe78d2bbe8de27e56d2be0acceaecfaa608ec132732189a8d0; record id: protein-stability-20261009-result-chu2024-rosetta-mega-scale-spearman-correlation; fields: status; attributes.metric; attributes.metric_qualifier; attributes.printed_value; attributes.numeric_value; sha256: f6bbc25f6760c82aba00ad40db1869002ccd85a00ef60fc81e150df4f383ca41; record id: protein-stability-20261009-source-chu2024; fields: status; attributes.artifact_sha256; sha256: e959594a702e1cfb9147c235f9a31116ada6106293e06b6e23185a14a95803b0; record id: use-case-protein-stability; fields: status; name; description; facets; attributes.question; attributes.decision; attributes.inputs; attributes.output; attributes.exclusions; attributes.setting; attributes.clinical_scope; attributes.intended_users; sha256: 2dd1d0e139a17e51ae72cc5945196ffbb366fe4daa4c20c4efd950346a30a54f
Related records

Suggest a correction