FEP+ benchmark four-target subset, affinity ranking and cost

Stable ID: uc-leadopt-20261022-protocol-boltz2-fep-4target

areas

molecular-interactions

contexts

research

protocol

Per-method compute time plus Pearson R, Kendall tau, PMAE, MAE (non-centered and centered) and the fraction of predictions within 1 and 2 kcal/mol, as printed in Table 12

version

Boltz-2 supplement (bioRxiv v1) Table 12

aggregation

Calculated per assay and averaged across assays with weight proportional to the number of compounds in each assay (appendix D.2.2, 'Evaluation metrics'). The Pearson R, Kendall tau and PMAE columns are printed under the label 'target avg.'; assays and targets are not the same unit, and the source does not reconcile the printed label with the stated computation.

metric definition

Appendix D.2.2 'Evaluation metrics': Pearson R, Kendall tau, pairwise MAE (PMAE), MAE and the fraction of predictions within 1 and 2 kcal/mol, each computed per assay and averaged across assays weighted by compound count, with predictions converted to kcal/mol first; the centered variants translate each assay's predictions to the ground-truth mean; the 'time' column prints a per-method cost with no stated denominator

control

Training proteins with at least 90% sequence similarity to a validation or test protein were removed, by clustering all affinity-dataset protein sequences with 'mmseqs easy-cluster ... --min-seq-id 0.9 --cov-mode 0 -c 0.01' and dropping any training protein sharing a cluster with a validation or test protein (appendix D.2.1, 'Data leakage control'). Compound-level dependence was checked in Figure 10 and no strong dependence was observed; that figure prints no values.

heldout test

Held-out affinity test set; training proteins sharing an mmseqs 90% identity cluster with a test protein were removed

source locator

Table 12, '4 target subset of the FEP+ benchmark'

limitations

One value per cell with no uncertainty; bootstrap standard errors appear only as error bars in Figure 6, which prints no numbers.; The column headers read 'target avg.' while appendix D.2.2 defines the aggregation as a compound-weighted mean across assays. Assays and targets are not the same unit and the source does not reconcile the two labels.; No per-assay or per-series value is printed, so the table supports statements about the mean over series and not about any one series.; The units of the PMAE and MAE columns are not printed in the table header; appendix D.2.2 states that all predictions are converted to kcal/mol before the metrics are computed.; The 'Perc. within 1' and 'Perc. within 2' headers say percentage, but appendix D.2.2 defines them as the fraction of predictions with absolute error below the threshold and the printed values lie on a 0 to 1 scale.; The results of FEP+ are intended to show the maximum attainable accuracy of current (commercial) FEP simulations by manually adjusting the protocol to the system at hand after observing the error with respect to the experiment, so the FEP+ arm is not a prospective baseline.; The supplement does not state whether the OpenFE, FEP+ and ABFE rows were re-run here or taken from the cited prior work, so the origin of those evaluations is recorded as unreported.; The 'time' column does not say what it counts: whether it is per compound, per edge, per target or for the whole subset, nor on what hardware. The FEP+ cell is printed as '-', so the table carries no measured FEP+ cost.; Four of the ten rows print the accuracy columns beyond Pearson R and Kendall tau as N/A, so the arms are not all comparable on the same metrics.; This subset and the 876-complex OpenFE subset both derive from the same collection and must partly overlap; the two tables are not independent.; The series definition behind the 87 compounds rests on a citation ('Chen et al., 2023') that does not resolve in the supplement's reference list.; One target can carry several assays, so a compound-weighted mean across assays is not a mean over targets even where the two happen to coincide; Ross et al. 2023 Table 1 surveys this collection's own proteins measured in more than one assay.

missing metadata

uncertainty reporting: reason: unreported; note: Table 12 prints no uncertainty

review

method: source-hash-verification; pdf-text-parse; independent-cell-check; rendered-source-visual-check; ai-assisted-source-review; method note: Re-retrieved all three sources and matched each pinned SHA-256. Re-rendered the Boltz-2 supplement with pdftotext -layout (poppler 26.08.0) and matched the text layer's recorded SHA-256. Re-read all 256 cells with a parser written for this review that does not import the extraction script, matching each result to its cell through its own source_locator, and separately read appendix tables 11, 12 and 14 from PNG renderings of PDF pages 29 and 30 at 160 dpi, and Ross Table 3 from the pinned XML markup. Checked printed value, numeric value, locator, metric concept, qualifier, direction, unit, uncertainty and the linked configuration, protocol and dataset, and that every N/A or '-' cell is recorded as missing with a reason.; reviewer: claude; reviewer note: Claude review agent, independent of the extraction agent; no human review claimed; reviewed at: 2026-10-22T13:40:00Z; artifact sha256: be867a5d92add8b223d47d882a4213ad489741f3e4d5081a0901a54370342569; retrieval url: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC12262699/supplementaryFiles

• part of: Public binding free energy benchmark (Ross et al. 2023)
• uses data: Four-target subset of the public FEP benchmark
• evaluates task: Small-molecule affinity
• subject: Leakage control applied to the four-target subset
• subject: The speed claim against FEP is not backed by a measured FEP cost
• assessment: ABFE on the FEP+ benchmark four-target subset
• assessment: BACPI on the FEP+ benchmark four-target subset
• assessment: Boltz-2 on the FEP+ benchmark four-target subset
• assessment: Boltz-2 iptm on the FEP+ benchmark four-target subset
• assessment: Chemgauss4 on the FEP+ benchmark four-target subset
• assessment: FEP+ on the FEP+ benchmark four-target subset
• assessment: FMO on the FEP+ benchmark four-target subset
• assessment: GAT on the FEP+ benchmark four-target subset
• assessment: MM/PBSA on the FEP+ benchmark four-target subset
• assessment: OpenFE on the FEP+ benchmark four-target subset
• assessed by: Rank close analogues of a hit by binding affinity