Assessment of protein-ligand pose and affinity predictions in CASP16

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 1 benchmark
• 1 dataset
• 7 evidence claims
• 1 use case

Claims

Statements this source supports, each with where in the source it was checked.

• Achievable ranking accuracy under a larger assumed assay error: Finding:ceiling-2kcal mean expected best Kendall's τ of 0.43 for autotaxin at 2 kcal/mol assumed error. Source wording: "Thus, if we assign an error of 30× in IC50 (2 kcal/mol in binding free energy), we obtain a mean expected best Kendall's τ of 0.43 for autotaxin, which is similar to the best result in Figure 8A." (Section 3.3.1)
• CASP16 affinity accuracy is no better than the earlier D3R challenges: Finding:no-better-than-d3r best CASP16 stage 1 results no better than the typical best stage 1 D3R result, which averaged Kendall's τ of 0.48 (min 0.21, max 0.71). Source wording: "For affinity predictions, it seems clear that the best Stage 1 CASP16 results are no better than the typical best Stage 1 D3R result, which averaged Kendall's τ of 0.48, with min/max of 0.21 and 0.71." (Discussion)
• Estimated achievable ranking accuracy under assay error: Finding:achievable-ceiling expected mean best Kendall's τ of 0.73. Source wording: "Weighting these values by the number of measurements in each dataset, as done in preparing the summary data in Figure 9A, we obtain an expected mean best Kendall's τ of 0.73." (Section 3.3.1)
• Evidence summary: ranking analogues of a hit by affinity: Summary Three sources carry the extracted evidence. Every number below is a mean over congeneric series: each error and correlation is computed inside one series first, so these are averages of within-series values and not correlations pooled across unrelated targets. No source prints a value for an individual series, or the spread across series. Ross et al. 2023, written by the developers of the FEP+ workflow, sets both the free energy baseline and the assay noise floor on the same page. On a curated collection of 1,237 compounds across more than 50 proteins, FEP+ reached a compound-weighted pairwise RMSE of 1.25 kcal/mol [1.17, 1.33], a pairwise mean unsigned error of 0.98 [0.91, 1.05], a coefficient of determination of 0.56 [0.51, 0.60] and a Kendall tau of 0.51 [0.48, 0.55]; its edgewise errors, which depend on the perturbation graph's topology, were 1.17 [1.08, 1.25] and 0.91 [0.84, 0.98]. The same table's survey of 1,867 compounds whose affinities were measured in at least two independent assays gives the noise floor: pairwise RMSE 0.91 kcal/mol [0.83, 1.11], mean unsigned error 0.67 [0.61, 0.83], R2 0.79 [0.75, 0.82] and Kendall tau 0.71 [0.65, 0.74]. So the error of the free energy calculation is close to the error between two assays, while its rank correlation sits well below. The FEP+ protocol was adjusted by hand per system after the error against experiment had been seen, so this is a maximum-attainable rather than a prospective figure. The Boltz-2 supplement, written by the developers of Boltz-2, is the only source here that puts a machine-learning predictor beside free energy calculation on the same compounds. On 876 protein-ligand complexes from the same collection, Boltz-2 reached an assay-weighted Pearson R of 0.62 and Kendall tau of 0.46, against 0.63 and 0.47 for OpenFE under a fixed automated protocol and 0.72 and 0.53 for FEP+; the sequence-only baselines reached 0.29 and 0.19 (BACPI) and 0.28 and 0.20 for its ligand-only ablation, and Boltz-2's own interface confidence score reached -0.07 and -0.05. On a four-target subset of 87 neutral compounds the supplement prints compute cost beside accuracy: Boltz-2 0.66 and 0.48 at 20 GPU seconds, OpenFE 0.66 and 0.51 at 6-12 GPU hours, ABFE 0.75 and 0.54 at more than 20 GPU hours, FEP+ 0.78 and 0.63 with its time cell printed as '-', FMO 0.55 and 0.38 at 2-10 CPU minutes, Chemgauss4 0.26 and 0.17 at 20-30 CPU seconds and MM/PBSA 0.18 and 0.16 at 10-15 CPU minutes. No measured FEP+ cost appears in the table. On the CASP16 blind challenge, where the mmseqs 90% identity filtering used for the other benchmarks was not applied because the data postdates the training cutoff, the supplement reports Boltz-2 at 0.65 and 0.45 against the highest-ranked participant at 0.54 and 0.42. Tables 11, 12 and 14 print one value per cell with no uncertainty. The independent CASP16 assessment sets those challenge figures against a ceiling. Its abstract gives the maximum Kendall tau across all participants as 0.42, against an estimated best achievable value of 0.73 derived by resampling the measurements under an assumed threefold IC50 error (0.54 with a standard deviation of 0.11 for chymase, 0.76 with 0.02 for autotaxin); under an assumed 30-fold error the expected best tau for autotaxin falls to 0.43. Its body text gives the maximum achieved value as 0.39 for one figure and 'about 0.4' elsewhere without reconciling the three numbers, and it scores submissions under its own weighting and skip conventions, so its 0.42 and the 0.42 printed for group LG016 in the Boltz-2 supplement are not established to be the same quantity. Supplying experimental structures in stage 2 produced no shift toward greater accuracy, and the assessors note that the best CASP16 stage-1 results are no better than the typical best stage 1 D3R result, which averaged a Kendall tau of 0.48 with a minimum of 0.21 and a maximum of 0.71. Two comparisons set a machine-learning predictor against free energy calculation on the same series, and both are reported by the developers of the machine-learning method: the Boltz-2 supplement's two FEP+ benchmark subsets. Both take their FEP+ arm from a benchmark whose protocol was tuned per system after seeing the experimental error, which raises the baseline being compared against. And because every value is a mean over series, these numbers say how well each method ranked analogues on average across these series, not how well it would rank any one series. (Ross et al. 2023 Tables 1-3; Boltz-2 supplement Tables 11, 12 and 14; Gilson et al. 2026 Abstract and Section 3.3.1)
• Maximum affinity-ranking accuracy reached in CASP16: Finding:max-kendall-tau maximum Kendall's τ = 0.42 (abstract); the body text gives 0.39 for Figure 9A and 'about 0.4' elsewhere. Source wording: "The CASP affinity predictions showed modest correlation with experimental data (maximum Kendall's τ = 0.42), well below the theoretical maximum possible given experimental uncertainty (~0.73)." (Abstract; compare Section 3.3.1, 'This is well above the maximum achieved value of 0.39 (Figure 9A)' and 'the maximum value of Kendall's tau observed here, about 0.4')
• Per-target achievable ranking accuracy under assay error: Finding:ceiling-per-target chymase 0.54 (SD 0.11), autotaxin 0.76 (SD 0.02). Source wording: "For chymase, the mean resampled Kendall's τ is 0.54 (SD: 0.11), and for autotaxin, the value is 0.76 (SD: 0.02)." (Section 3.3.1)
• Supplying experimental structures did not improve affinity accuracy: Finding:structures-did-not-help no shift toward greater accuracy in stage 2. Source wording: "It is evident from the histograms of Kendall's τ values over all groups (Figure 8B,C) that knowledge of the structures did not lead to a shift toward greater accuracy." (Section 3.3.1)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.