- You bring
- Wild-type protein sequence and amino acid substitutions
- A defined construct and experimental stability endpoint; alignment-derived methods additionally require a traceable alignment and model
- You want
- Per-method correlation, error and antisymmetry values from the linked comparisons, each with its dataset, split and origin, and the exact existing AMFR configurations with their assay scope; unresolved comparisons are listed. No universal model ranking or pathogenicity prediction.
- Evidence in this release
- 7 comparisons shown · 1 held for review · 68 tools · 141 configurations
- Kept up to date
- Evidence last checked 14 October 2026 · re-checked monthly, next due 14 November 2026
For "Assess methods for protein stability experiments", on S669 external ddG benchmark (Pancotti et al. 2022), Pearson (S669, 669 direct variants) (Pearson), MAESTRO scored 0.50, reported by Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset.
Best reported valueMAESTROPredicting protein stability changes upon single-point…
Pearson0.50S669 external ddG benchmark (Pancotti et al. 2022), Pearson (S669, 669 direct variants)
Baseline on this protocolNone scoredNo margin is computed
Ranked tools2113 from independent studies
21 tools have a value on this protocol; 13 of them come from studies whose authors did not develop the tool. No baseline method is scored on this protocol.
Ranking
Ranked on S669 external ddG benchmark (Pancotti et al. 2022), Pearson (S669, 669 direct variants) (Pearson, higher is better). Only results on this protocol are ranked; other tools are listed below the table.
16 tools in 21 rows, from 1 source. 13 of 21 results (62%) come from an independent study; 8 are reported by the tool's developers.
52 tools are evaluated elsewhere in this use case but not on this protocol, so not ranked
Comparisons
Unified model comparison for this use case. Compare tools against protocol baselines in a single view.
Rows marked Contested rest on results that other studies disagree with. The results are still shown. See the disagreements that involve this use case for what disagrees and what we are doing about it.
S669 external ddG benchmark (Pancotti et al. 2022)
Direct evidenceIndependent study: Reported in a study whose authors did not develop the evaluated tool.Developer-reported: Reported by the developers of the evaluated tool.Pearson r, RMSE and MAE of predicted against experimental ddG for 21 predictors on 669 direct and 669 reverse variants, with antisymmetry
Sorted by Pearson (higher is better). Showing the top 10 of 16 rows. The best value in each column is highlighted.
| Tool | Pearson (S669, 669 direct variants) | Pearson (S669, 669 reverse variants) | Pearson (S669, all 1,338 direct and reverse variants) | Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better)) | Antisymmetry bias (S669, antisymmetry bias) | MAE (S669, 669 direct variants) | MAE (S669, 669 reverse variants) | MAE (S669, all 1,338 direct and reverse variants) | RMSE (S669, 669 direct variants) | RMSE (S669, 669 reverse variants) | RMSE (S669, all 1,338 direct and reverse variants) |
|---|
| 0.50 (best in column) | 0.20 | 0.44 | –0.22 | –0.57 | 1.06 kcal/mol | 1.655 kcal/mol | 1.30 kcal/mol | 1.44 kcal/mol (best in column) | 2.10 kcal/mol | 1.80 kcal/mol |
|---|
| 0.46 | 0.45 (best in column) | 0.61 | –0.98 | –0.02 | 1.05 kcal/mol (best in column) | 1.060 kcal/mol | 1.05 kcal/mol (best in column) | 1.49 kcal/mol | 1.50 kcal/mol | 1.50 kcal/mol |
|---|
| 0.43 | 0.41 | 0.57 | –0.97 | –0.05 | 1.11 kcal/mol | 1.140 kcal/mol | 1.13 kcal/mol | 1.60 kcal/mol | 1.62 kcal/mol | 1.61 kcal/mol |
|---|
| 0.43 | 0.43 | 0.61 | –1 (best in column) | 0.00 kcal/mol | 1.09 kcal/mol | 1.100 kcal/mol | 1.10 kcal/mol | 1.52 kcal/mol | 1.53 kcal/mol | 1.52 kcal/mol |
|---|
| 0.41 | 0.23 | 0.41 | –0.12 | –0.67 | 1.10 kcal/mol | 1.680 kcal/mol | 1.39 kcal/mol | 1.52 kcal/mol | 2.14 kcal/mol | 1.86 kcal/mol |
|---|
| 0.41 | 0.34 | 0.50 | –0.58 | –0.06 | 1.19 kcal/mol | 1.240 kcal/mol | 1.21 kcal/mol | 1.60 kcal/mol | 1.69 kcal/mol | 1.65 kcal/mol |
|---|
| 0.41 | 0.24 | 0.46 | –0.32 | –0.69 | 1.09 kcal/mol | 1.640 kcal/mol | 1.37 kcal/mol | 1.51 kcal/mol | 2.09 kcal/mol | 1.82 kcal/mol |
|---|
| 0.41 | 0.42 | 0.62 (best in column) | –0.85 | 0.09 kcal/mol | 1.08 kcal/mol | 1.050 kcal/mol (best in column) | 1.07 kcal/mol | 1.50 kcal/mol | 1.49 kcal/mol (best in column) | 1.49 kcal/mol (best in column) |
|---|
| 0.41 | 0.13 | 0.32 | –0.4 | –0.4 | 1.26 kcal/mol | 1.640 kcal/mol | 1.45 kcal/mol | 1.67 kcal/mol | 2.16 kcal/mol | 1.93 kcal/mol |
|---|
| 0.39 | 0.40 | 0.47 | –0.72 | –0.61 | 2.08 kcal/mol | 2.020 kcal/mol | 2.05 kcal/mol | 2.70 kcal/mol | 2.68 kcal/mol | 2.69 kcal/mol |
|---|
Show all 16 rowsShow the top 10 rows only
Caveats and method
- Single benchmark of 669 variants with reverse variants built from modelled structures.
- Several methods were developed by authors of this paper: ACDC-NN, ACDC-NN-Seq, DDGun and DDGun3D, and, through P. Fariselli and E. Capriotti, INPS3D, INPS-Seq, I-Mutant3.0 and I-Mutant3.0-Seq.
- Predictor versions and servers are listed only in the Supplementary Materials, which were not read; web servers may have changed since.
- Proteins are under 25% identity to S2648 and VariBench, but predictors trained on other data (for example ThermoMutDB itself) may still overlap.
- No uncertainty is printed.
- Direct-variant correlation alone misleads: MAESTRO has the highest direct Pearson (0.5) but a reverse Pearson of 0.2 and a bias of -0.57 (Table 1).
Protocol: S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)
Primary benchmark evaluation shown (S669 external ddG benchmark (Pancotti et al. 2022)). 6 additional study protocols (ProteinGym AMFR official zero-shot Spearman snapshot, Transfer from mega-scale domains to larger proteins (Chu et al. 2024), ThermoMPNN comparison sets (Dieckhaus et al. 2024) and others) are cited in individual tool drawer rows and under Question, sources and review.
Held for review
Recorded but not shown as evidence until the reason below is resolved.
DeepProtein Stability (TAPE): ρ ↑
Not shown: the source carries an evidence concern noting that Figure 3's caption calls the regression metric 'Spearman (Pearson) Coefficient' while the Evaluation metrics section defines it as Spearman. A source with evidence concerns withholds every judgement that cites it, so this judgement and the use-case summary edit that would describe it are held until the concern is cleared.
Not covered yet
- The existing ESM-2 and fixed-seed random results use separate protocols. They are displayed separately and do not establish a matched cross-protocol winner.
- No uncertainty intervals or seed-variability estimates are recorded for these completed evaluations. One random ranking is not a chance-performance interval.
- The proposed ESM-2 versus EVCouplings site-independent and EVmutation comparison targets a frozen set of covered single substitutions. It has no completed results.
- The planned comparison still needs a separate execution decision, a traceable real evolutionary model, the full-model adapter, resource logging, frozen populations and tested analysis code. Planning resource ceilings are not measured requirements.
- The recorded ESM-2 timer excludes checkpoint loading, preparation and metrics; peak memory is unreported. It is not an end-to-end or cross-model speed comparison.
- The existing AMFR protocols have no reviewed direct task-membership link. The mappings therefore reference the protocols only and do not infer membership from the ProteinGym suite.
- Assay bytes were hashed locally without independent authentication against an upstream published checksum. Training overlap, independent reproduction and human scientific review remain unresolved.
- 30 September 2026 audit: The official comparison contains 820 single and 2,152 double substitutions. It does not complete the planned matched singles-only local study.
- 30 September 2026 audit: Number of Mutants=2,972 is the assay count; exact scored count per model, source score transformations and model checkpoint hashes remain unextracted.
- 30 September 2026 audit: Method input modalities differ. These scores support short-construct folding-stability ranking, not a universal sequence-only winner, full-protein function or clinical pathogenicity.
- Both mega-scale comparisons are developer papers (ThermoMPNN, Dieckhaus et al. 2024; ESM therm, Chu et al. 2024); no independent comparison on the mega-scale proteolysis set with printed per-method values was found.
- None of the new comparisons reports a result for the AMFR construct of the existing judgements.
- Only Pancotti et al. 2022 tests antisymmetry across many tools, and its authors developed several of them; Dieckhaus et al. 2024 report Ssym inverse variants from compiled values, and Chu et al. 2024 do not test it.
- Transfer to larger proteins is tested only by Chu et al. 2024, on six single proteins of 177 to 501 residues with Spearman correlation only.
- No linked comparison reports uncertainty intervals for per-method values; Dieckhaus et al. 2024 Table 1 gives seed standard deviations for ThermoMPNN and ProteinMPNN only.
- Stability Oracle and ABYSSAL appear only as values compiled from their own preprints (Dieckhaus et al. 2024 Table 3), not as reruns.
- Dieckhaus et al. 2024 compile an ACDC-NN S669 RMSE (1.60) that disagrees with the source they cite (Pancotti et al. 2022: 1.49); the compiled value is not shown.
Contribute evidence or propose a correction
How to cite
Assess methods for protein stability experiments. Rewire biological model benchmark database, release 2026-11-21-adc15377fab2. https://benchmarks.rewirebio.io/use-cases/protein-variant-stability/
Release 2026-11-21-adc15377fab2. Rankings and values change between releases, so keep the release id in the citation. Cite the original publication as well for any value you reuse: each result links its source.
The compilation and annotations made by Rewire in this release are offered under CC BY 4.0; values and text taken from cited sources stay under the terms of those sources and are not covered by this licence. Data licence
Question, sources and review
Who this is for, setting and exclusions
Inspect per-method correlation and error on held-out or dissimilar proteins, the antisymmetry of predictions on reverse variants and the change from small domains to larger proteins, with the AMFR assay results as a narrow example, then identify the comparison and validation still required for the target protein and endpoint.
- Who this is for
- Experimental researchers selecting variants for protein stability experiments
- Computational researchers evaluating sequence-based variant rankings
- Translational researchers checking whether stability evidence answers a disease question
- Biological setting
- The AMFR evidence is the 47-residue AMFR_HUMAN_Tsuboyama_2023_4G3O construct in ProteinGym v1.3. Its cDNA-display proteolysis assay infers folding stability. Completed evaluations cover 2,972 variants: 820 single and 2,152 double substitutions. Published comparisons add single-point stability benchmarks on held-out or dissimilar proteins (S669, homology-split mega-scale and FireProtDB sets, held-out mega-scale domains) and six single-protein stability datasets of 177 to 501 residues.
- Clinical research scope
- Clinical applicability is not established. Stability prediction on benchmark proteins or on a short experimental construct is not evidence of clinical pathogenicity or suitability for diagnosis or treatment.
Outside this use case
- Whole-protein function, cellular activity, organismal fitness and clinical pathogenicity
- Transfer of a benchmark result to a different target protein, protein size or assay without validation on that protein
- Treating the existing mixed cohort as a completed matched single-substitution comparison
Related articles
Read the background, methods and practical considerations behind this question.
Background on protein variant scoring, assay-specific evaluation and the limits of transferring results between protein tasks.
Sources, review and downloads
Automated source review · 2026-09-30 · Codex source curation, independent worker cross-review and root integration review
Source-backed literature curation with independent automated transcription and scope checks. No new model execution, independent experimental replication, qualified human scientific review or clinical validation.
Release 2026-11-21-adc15377fab2 · question use-case-protein-stability
Use-case input digest bfbb7ab9303becad1e3cb395568c26f41e72ddb67cd1a89a06d9eaa0f76bd95f
Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)