The maximal and current accuracy of rigorous protein-ligand binding free energy calculations

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 2 methods
• 2 configurations
• 1 benchmark
• 2 protocols
• 2 datasets
• 2 evaluations
• 12 results
• 24 evidence claims
• 1 use case

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• Evidence summary: ranking analogues of a hit by affinity: Summary Three sources carry the extracted evidence. Every number below is a mean over congeneric series: each error and correlation is computed inside one series first, so these are averages of within-series values and not correlations pooled across unrelated targets. No source prints a value for an individual series, or the spread across series. Ross et al. 2023, written by the developers of the FEP+ workflow, sets both the free energy baseline and the assay noise floor on the same page. On a curated collection of 1,237 compounds across more than 50 proteins, FEP+ reached a compound-weighted pairwise RMSE of 1.25 kcal/mol [1.17, 1.33], a pairwise mean unsigned error of 0.98 [0.91, 1.05], a coefficient of determination of 0.56 [0.51, 0.60] and a Kendall tau of 0.51 [0.48, 0.55]; its edgewise errors, which depend on the perturbation graph's topology, were 1.17 [1.08, 1.25] and 0.91 [0.84, 0.98]. The same table's survey of 1,867 compounds whose affinities were measured in at least two independent assays gives the noise floor: pairwise RMSE 0.91 kcal/mol [0.83, 1.11], mean unsigned error 0.67 [0.61, 0.83], R2 0.79 [0.75, 0.82] and Kendall tau 0.71 [0.65, 0.74]. So the error of the free energy calculation is close to the error between two assays, while its rank correlation sits well below. The FEP+ protocol was adjusted by hand per system after the error against experiment had been seen, so this is a maximum-attainable rather than a prospective figure. The Boltz-2 supplement, written by the developers of Boltz-2, is the only source here that puts a machine-learning predictor beside free energy calculation on the same compounds. On 876 protein-ligand complexes from the same collection, Boltz-2 reached an assay-weighted Pearson R of 0.62 and Kendall tau of 0.46, against 0.63 and 0.47 for OpenFE under a fixed automated protocol and 0.72 and 0.53 for FEP+; the sequence-only baselines reached 0.29 and 0.19 (BACPI) and 0.28 and 0.20 for its ligand-only ablation, and Boltz-2's own interface confidence score reached -0.07 and -0.05. On a four-target subset of 87 neutral compounds the supplement prints compute cost beside accuracy: Boltz-2 0.66 and 0.48 at 20 GPU seconds, OpenFE 0.66 and 0.51 at 6-12 GPU hours, ABFE 0.75 and 0.54 at more than 20 GPU hours, FEP+ 0.78 and 0.63 with its time cell printed as '-', FMO 0.55 and 0.38 at 2-10 CPU minutes, Chemgauss4 0.26 and 0.17 at 20-30 CPU seconds and MM/PBSA 0.18 and 0.16 at 10-15 CPU minutes. No measured FEP+ cost appears in the table. On the CASP16 blind challenge, where the mmseqs 90% identity filtering used for the other benchmarks was not applied because the data postdates the training cutoff, the supplement reports Boltz-2 at 0.65 and 0.45 against the highest-ranked participant at 0.54 and 0.42. Tables 11, 12 and 14 print one value per cell with no uncertainty. The independent CASP16 assessment sets those challenge figures against a ceiling. Its abstract gives the maximum Kendall tau across all participants as 0.42, against an estimated best achievable value of 0.73 derived by resampling the measurements under an assumed threefold IC50 error (0.54 with a standard deviation of 0.11 for chymase, 0.76 with 0.02 for autotaxin); under an assumed 30-fold error the expected best tau for autotaxin falls to 0.43. Its body text gives the maximum achieved value as 0.39 for one figure and 'about 0.4' elsewhere without reconciling the three numbers, and it scores submissions under its own weighting and skip conventions, so its 0.42 and the 0.42 printed for group LG016 in the Boltz-2 supplement are not established to be the same quantity. Supplying experimental structures in stage 2 produced no shift toward greater accuracy, and the assessors note that the best CASP16 stage-1 results are no better than the typical best stage 1 D3R result, which averaged a Kendall tau of 0.48 with a minimum of 0.21 and a maximum of 0.71. Two comparisons set a machine-learning predictor against free energy calculation on the same series, and both are reported by the developers of the machine-learning method: the Boltz-2 supplement's two FEP+ benchmark subsets. Both take their FEP+ arm from a benchmark whose protocol was tuned per system after seeing the experimental error, which raises the baseline being compared against. And because every value is a mean over series, these numbers say how well each method ranked analogues on average across these series, not how well it would rank any one series. (Ross et al. 2023 Tables 1-3; Boltz-2 supplement Tables 11, 12 and 14; Gilson et al. 2026 Abstract and Section 3.3.1)
• FEP+ protocol tuned per system after seeing the error against experiment: Limitation The Boltz-2 supplement states that 'The results of FEP+ are intended to show the maximum attainable accuracy of current (commercial) FEP simulations by manually adjusting the protocol (input preparation, perturbation map, force field) to the system at hand after observing the error with respect to the experiment [Ross et al., 2023]'. Ross et al. 2023 describes the same adjustments in its own words: extra rotamer states, protonation states, atom mappings and scaled torsions were added 'from visual inspection and manual intervention' when poor sampling was observed in prior simulations. (Boltz-2 supplement appendix D.2.2, 'Baselines'; Ross et al. 2023 Results, 'Using FEP to resolve ambiguities in the structural inputs')
• Per-series affinity-ranking values are not printed: Limitation Every value in Boltz-2 supplement Tables 11, 12 and 14 and in Ross et al. 2023 Table 3 is an average over series: the Boltz-2 metrics are computed per assay and averaged across assays weighted by compound count, and the Ross values are weighted averages over FEP graphs. Neither source prints a per-series value, and neither prints the spread across series, so these tables support statements about the mean over series and not about any one series. (Boltz-2 supplement appendix D.2.2 'Evaluation metrics' and Tables 11, 12, 14; Ross et al. 2023 Table 3 legend)
• Public FEP benchmark composition: Bayer macrocycles54: Composition:bayer-macrocycles54 Ftase, BRD4; 8 compounds (Table 2, row "Bayer macrocycles54")
• Public FEP benchmark composition: FEP+ charge-change55: Composition:fep-plus-charge-change55 CDK2, DLK, EGFR, EPHX2, IRAK4, ITK, JAK1, JNK1, PTP1B, TYK2; 53 compounds (Table 2, row "FEP+ charge-change55")
• Public FEP benchmark composition: FEP+ Fragments57: Composition:fep-plus-fragments57 T4 lysozyme, LigA, Mcl1, MUP-1, JAK-2, hsp90, p38; 79 compounds (Table 2, row "FEP+ Fragments57")
• Public FEP benchmark composition: FEP+ macrocycles31: Composition:fep-plus-macrocycles31 BACE1, CHK1, CK2, MHT1, HSP90; 34 compounds (Table 2, row "FEP+ macrocycles31")
• Public FEP benchmark composition: FEP+ R-group set16: Composition:fep-plus-r-group-set16 BACE1, CDK2, JNK1, Mcl1, p38, PTP1B, thrombin, TYK2; 199 compounds (Table 2, row "FEP+ R-group set16")
• Public FEP benchmark composition: FEP+ scaffold-hopping32: Composition:fep-plus-scaffold-hopping32 BACE1, β-tryptase, CHK1, ERα, Factor Xa,; 17 compounds (Table 2, row "FEP+ scaffold-hopping32")
• Public FEP benchmark composition: GPCRs74,75: Composition:gpcrs74-75 A2A, OX2, P2Y1; 98 compounds (Table 2, row "GPCRs74,75")
• Public FEP benchmark composition: Janssen BACE136,76: Composition:janssen-bace136-76 BACE1; 74 compounds (Table 2, row "Janssen BACE136,76")
• Public FEP benchmark composition: MCS docking77: Composition:mcs-docking77 HNE, Renin; 49 compounds (Table 2, row "MCS docking77")
• Public FEP benchmark composition: Merck sets56: Composition:merck-sets56 CDK8, cMet, Eg5, HIF-2α, PFKFB3, SHP-2, SYK, TNKS2; 264 compounds (Table 2, row "Merck sets56")
• Public FEP benchmark composition: Miscellaneous: Composition:miscellaneous CDK878, Galectin10,79, BTK80, HIV1 protease81, FAAH82; 79 compounds (Table 2, row "Miscellaneous")
• Public FEP benchmark composition: OPLS drug discovery27: Composition:opls-drug-discovery27 A, B, C, D, E; 93 compounds (Table 2, row "OPLS drug discovery27")
• Public FEP benchmark composition: OPLS stress set27: Composition:opls-stress-set27 BACE1, CHK1, Factor Xa; 114 compounds (Table 2, row "OPLS stress set27")
• Public FEP benchmark composition: Total: Composition:total 1237 compounds (Table 2, row "Total")
• Public FEP benchmark composition: Water displacement34: Composition:water-displacement34 BRD4(1), CHK1, Hsp90, scytalone dehydratase, TAF1(2), thrombin, urokinase; 76 compounds (Table 2, row "Water displacement34")
• Relevance of Public FEP benchmark to ranking analogues of a hit: Assessed by Public FEP benchmark, FEP+ accuracy against experiment (Tables 1, 2 and 3; Methods 'Accuracy metrics'; Results 'Using FEP to resolve ambiguities in the structural inputs')
• Relevance of Repeat-assay ceiling to ranking analogues of a hit: Assessed by Repeat-assay reproducibility of relative binding free energies (Tables 1, 2 and 3; Methods 'Accuracy metrics'; Results 'Using FEP to resolve ambiguities in the structural inputs')
• Repeat-assay survey composition: Binding vs binding: Composition:binding-vs-binding SH2, Herg, CaI, CaII, Lectin, HCV polymerase, FAK, trypsin, DPPIV, bromodomains, BPTF, galectin 3; 250 compounds (Table 1, row "Binding vs binding")
• Repeat-assay survey composition: Binding vs functional: Composition:binding-vs-functional Xiap, thrombin, HCV polymerase, Hsp90, FAK, DPPIV, MAPK13, AChE; 1207 compounds (Table 1, row "Binding vs functional")
• Repeat-assay survey composition: Functional vs functional: Composition:functional-vs-functional COT kinase, ACE, SARS-Cov2 main protease; 410 compounds (Table 1, row "Functional vs functional")
• Repeat-assay survey composition: Total: Composition:total 1867 compounds (Table 1, row "Total")

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.