rewirebio.iobenchmarks

Use caseProteins and complexesResearch only

Choose structural hypotheses to guide experiments

Which predicted interfaces or structures are reliable enough to guide my next experiment?

What the evidence shows

Two new comparisons join the existing docking and FoldBench evidence, both on antibody complexes, and neither ties a prediction to an experiment's outcome.

Fromm et al. 2026 generated 200 AlphaFold3 models for each of 110 antibody-antigen complexes deposited after the training cutoff, then asked which model each confidence score picks. Every score a user can compute picked models of about the same quality: mean DockQ 0.375 for ranking confidence, 0.370 for ipTM, 0.369 for pTM, 0.353 for ipSAE and 0.352 for pDockQ2, while the best of the 200 models averaged 0.544. The per-target correlation between score and accuracy stayed at or below 0.247, so within one target the confidence ranking carried little information.

Smorodina et al. 2026, a bioRxiv preprint that has not been peer reviewed, ran AlphaFold3, Boltz-2, Boltz-1 and Chai-1 on 106 nanobody-antigen complexes. Sampling raised the best available model for every tool, with median best DockQ rising from 0.24 to 0.68 for AlphaFold3 and from 0.57 to 0.80 for Boltz-2 between 1 and 100 samples, though picking that best sample needs the experimental structure. Confidence tracked accuracy unevenly: the correlation between ipTM and best-sample DockQ was 0.736 for AlphaFold3, 0.665 for Boltz-2 and 0.612 for Chai-1, with 18% of Boltz-2 predictions confident but wrong and 24% of Chai-1 predictions accurate but unconfident.

Pairing every nanobody with every antigen showed that confidence does not indicate that an interaction exists at all: average precision for ranking the real pairings above shuffled ones was 0.187 for AlphaFold3, 0.067 for Chai-1 and 0.026 for Boltz-2, against a baseline of about 0.011. The preprint's nanobody set mixes systems inside and outside each tool's training data, and its non-cognate pairings are assumed not to bind rather than tested. A high confidence score is a statement about a predicted structure, not evidence that two molecules interact.

You bring
  • Protein sequences, partner identities and the intended assembly
  • Available experimental structures or templates with release dates
  • A defined experimental choice and testing budget
  • Relevant biochemical conditions and alternative conformations
You want
Structural hypotheses with confidence and coverage estimates, linked to proposed mutations or constructs and tests of their reliability.
Evidence in this release
9 comparisons shown · 2 held for review · 10 tools · 21 configurations
Kept up to date
Evidence last checked 9 October 2026 · re-checked monthly, next due 9 November 2026
Executive Takeaway

For "Choose structural hypotheses to guide experiments", on Lipid-protein pose predictions, Lipid poses on 36 post-cutoff complexes (LiPP) (Ligand RMSD under 2a rate, a proxy protocol), AlphaFold 3 scored 47.2% {30.4–64.5}, reported by The LiPP Benchmark Set for Modeling Lipid–Protein Complexes: Comparison of Co-Folding and Docking Methods.

Best reported valueAlphaFold 3The LiPP Benchmark Set for Modeling Lipid–Protein…
Ligand RMSD under 2a rate47.2% {30.4–64.5}Lipid-protein pose predictions, Lipid poses on 36 post-cutoff complexes (LiPP)
Baseline on this protocolAutoDock Vina30.5% {16.3–48.1}
Ranked tools55 from independent studies

5 tools have a value on this protocol; 5 of them come from studies whose authors did not develop the tool. The use case's baseline, AutoDock Vina, scored 30.5% {16.3–48.1} on the same protocol. Reviewers judged this protocol a proxy for the use case.

Ranking

Ranked on Lipid-protein pose predictions, Lipid poses on 36 post-cutoff complexes (LiPP) (Ligand RMSD under 2a rate, higher is better). Proxy protocol Reviewers judged this protocol a proxy for the use case, not a direct test of it. Only results on this protocol are ranked; other tools are listed below the table.

5 tools, from 1 source. 5 of 5 results (100%) come from an independent study.

Rank▲ModelLigand RMSD under 2a rate (higher is better)vs AutoDock VinaEvidenceCoverageReported byLicense
#147.2% {30.4–64.5}+54.8%Source-checked Independent study5/9The LiPP Benchmark Set for Modeling…Custom AlphaFold 3 Model…
#236.1% {20.8–53.7}+18.4%Source-checked Independent study5/9The LiPP Benchmark Set for Modeling…Apache-2.0; README Licence…
#3
AutoDock Vina[-] Baseline
30.5% {16.3–48.1}Source-checked Independent study1/9The LiPP Benchmark Set for Modeling…Apache-2.0
#330.5% {16.3–48.1}0.0%Source-checked Independent study1/9The LiPP Benchmark Set for Modeling…MIT; README License…
#511.1% {3.1–26.0}-63.6%Source-checked Independent study1/9The LiPP Benchmark Set for Modeling…Not recorded
5 tools are evaluated elsewhere in this use case but not on this protocol, so not ranked

Baseline

AutoDock Vina and ClusPro docking. AutoDock Vina for ligands and ClusPro rigid-body docking for protein pairs are the established physics-based docking tools that co-folding models are compared with. Both sample poses and rank them with energy-based scoring functions; Vina's few scoring weights were fitted to PDBbind, but neither is a learned structure model.

Machine-learning methods are compared with the baseline on the same data in one table: Lipid-protein pose predictions. Baseline rows are marked in each table.

Tools compared

Comparisons

Unified model comparison for this use case. Compare tools against protocol baselines in a single view.

Lipid-protein pose predictions

Proxy evidenceIndependent study: Reported in a study whose authors did not develop the evaluated tool.

Ligand RMSD under 2a rate (higher is better), with the other metrics below each value. The best value in each column is highlighted.

ToolLipid poses on 36 post-cutoff complexes (LiPP)Lipid poses on 36 post-cutoff complexes (LiPP)
AlphaFold 3foundation model
47.2% {30.4–64.5} (best in column)Not reported
Chai-1foundation model
36.1% {20.8–53.7}Not reported
AutoDock VinaBaselineconventional
30.5% {16.3–48.1}Not reported
DiffDock-Lmachine learning
30.5% {16.3–48.1}Not reported
RoseTTAFold AAfoundation model
11.1% {3.1–26.0}Not reported
Caveats and method

What each column measures:

  • Lipid poses on 36 post-cutoff complexes (LiPP): Share of 36 post-cutoff lipid-protein complexes whose predicted lipid pose counts as a success (lipid pose all-atom RMSD below 2 Angstrom)
  • Lipid poses on 36 post-cutoff complexes (LiPP): Share of 36 post-cutoff lipid-protein complexes whose predicted lipid pose counts as a success (RMSD below 2 Angstrom and PoseBusters valid)
  • Author-run predictions, not an independent reproduction by Rewire.
  • The PoseBusters columns in the same tables are literature values compiled by the LiPP authors and are recorded as paper_compilation.
  • Pose accuracy only; the paper's confidence-score analysis is in figures and was not extracted.
  • Several printed test-set percentages are truncated rather than rounded (for example 11 of 36 printed as 30.5); the values are kept as printed.
  • The full LiPP set and the PoseBusters columns are not linked to the use case: the full set includes complexes inside some tools' training windows, and the PoseBusters values are literature compilations on drug-like ligands.

Protocols: LiPP Table 2: lipid pose all-atom RMSD below 2 Angstrom, LiPP test set, LiPP Table 3: RMSD below 2 Angstrom and PoseBusters valid, LiPP test set

Primary benchmark evaluation shown (Lipid-protein pose predictions). 8 additional study protocols (FoldBench protein-protein published full-set assessment, Antibody-antigen interface predictions, Protein-ligand pose predictions and others) are cited in individual tool drawer rows and under Question, sources and review.

Held for review

Recorded but not shown as evidence until the reason below is resolved.

Protein-protein interface predictions (DockQ classes on 994 post-2021 dimers (Mahtha et al. 2026))

Not shown: the source's experimental-method, template-quality and row-total cells conflict with its own text in places, recorded as evidence concerns on the source. This judgement uses only the 'No.' column, which agrees with the text, but a source with evidence concerns withholds every judgement that cites it until the concerns are resolved.

Monomer structure predictions (Predictive classes on 1666 post-2021 monomers (Mahtha et al. 2026))

Not shown: the source's experimental-method, template-quality and row-total cells conflict with its own text in places, recorded as evidence concerns on the source. This judgement uses only the 'No.' column, which agrees with the text, but a source with evidence concerns withholds every judgement that cites it until the concerns are resolved.

Not covered yet

  • Automated source review only; independent human scientific review remains outstanding.
  • No intervals in Table 3; no prospective experimental utility or matched conventional mutation/construct baseline.
  • Scored target populations differ; Table 1 assessable counts are retained as coverage context, not verified metric denominators. Some Table 3 rates do not reconcile with integer counts after rounding. No complete-cohort estimate is inferred.
  • ClusPro uses component 3D structures and BM5 targets, so scores cannot be directly compared with FoldBench; BM4 parameter benchmarking limits holdout independence. Exact ClusPro revision and input hashes remain unextracted.
  • ClusPro Total/easy/top10 source prints 87 although subtotals and 51.68% imply 77; literal 87 is retained as needs_review outside use-case mappings. Antibody and aggregate classes are not mapped to this issue.
  • Confidence selection is tested only for AlphaFold3 on antibody-antigen complexes (Fromm et al. 2026); no equivalent table was found for Boltz, Chai-1, protein-protein or protein-ligand complexes.
  • The nanobody evidence is a preprint that has not been peer reviewed, pools systems inside and outside each tool's training data, and defines its negatives as untested shuffled pairings.
  • No linked comparison reports uncertainty intervals for per-tool values.
  • No linked comparison connects a predicted structure or confidence score to the outcome of an experiment.
  • No machine-learning method has been compared with ClusPro docking on the same protein-protein data in the linked evidence.

Contribute evidence or propose a correction

How to cite

Choose structural hypotheses to guide experiments. Rewire biological model benchmark database, release 2026-11-21-c8bea25e5abb. https://benchmarks.rewirebio.io/use-cases/structural-hypotheses-experiments/

Release 2026-11-21-c8bea25e5abb. Rankings and values change between releases, so keep the release id in the citation. Cite the original publication as well for any value you reuse: each result links its source.

The compilation and annotations made by Rewire in this release are offered under CC BY 4.0; values and text taken from cited sources stay under the terms of those sources and are not covered by this licence. Data licence

Question, sources and review

Who this is for, setting and exclusions

Choose structural hypotheses and associated mutations or constructs for experimental testing.

Who this is for
  • Structural biologists
  • Researchers selecting interface mutations or protein constructs
Biological setting
Research planning within a bounded class of protein complexes. Monomers, antibody complexes and ligand complexes require separate comparison protocols.
Research setting
Research only. Structural hypotheses do not establish therapeutic efficacy or clinical suitability.

Outside this use case

  • Structural accuracy alone does not establish that a proposed experiment will succeed.
  • High confidence does not prove interaction existence, affinity or function.
  • Results from one complex class do not establish performance in another.
  • Confidence scores do not separate real from non-cognate pairings: on nanobody-antigen complexes the best average precision was 0.187 against a 0.011 baseline, so a confident model is not evidence that the interaction exists.
Evidence collection plan
Evidence collection planCollecting evidence

Already on record: 9 comparisons shown · 2 held for review · 10 tools, in the evaluated evidence above. The status above describes only the specific comparison in this plan, which remains open; it does not mean no evidence has been collected. The plan defines a comparison to investigate; it does not establish model performance or suitability.

Comparison question

Do model-derived structural hypotheses improve experimental choices over appropriate conventional strategies at the same testing budget?

Baselines, outcomes and validation requirements

Baselines to include

  • Suitable template modelling or docking for the specified complex class
  • Conventional mutation or construct selection for experimental-utility comparisons

Outcomes to measure

  • Task-specific contact or pose accuracy and prediction failures as intermediate evidence
  • Confidence calibration and coverage
  • Independent interface-mutation or construct success where measured

Validation requirements

  • Define the experimental choice and compatible complex class before selecting metrics.
  • Audit sequence, template, ligand and structure-deposition overlap with training data.
  • Include failed predictions and relevant alternative conformations.
  • Keep structural accuracy, interaction existence, affinity, function and experimental utility distinct.

Next collection task

Close the remaining endpoint, independence and comparator gaps documented in the 30 September 2026 audit and linked article issue; obtain qualified scientific review.

Related articles
Sources, review and downloads

Automated source review · 2026-09-30 · Codex source curation, independent worker cross-review and root integration review

Source-backed literature curation with independent automated transcription and scope checks. No new model execution, independent experimental replication, qualified human scientific review or clinical validation.

Release 2026-11-21-c8bea25e5abb · question use-case-structural-hypotheses-experiments

Use-case input digest bfbb7ab9303becad1e3cb395568c26f41e72ddb67cd1a89a06d9eaa0f76bd95f

Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)