FEP+ benchmark OpenFE subset, affinity ranking

Stable ID: uc-leadopt-20261022-protocol-boltz2-fep-openfe-subset

areas

molecular-interactions

contexts

research

protocol

Pearson R, Kendall tau, PMAE, MAE (non-centered and centered) and the fraction of predictions within 1 and 2 kcal/mol, as printed in Table 11

version

Boltz-2 supplement (bioRxiv v1) Table 11

aggregation

Calculated per assay and averaged across assays with weight proportional to the number of compounds in each assay (appendix D.2.2, 'Evaluation metrics'). The Pearson R, Kendall tau and PMAE columns are printed under the label 'target avg.'; assays and targets are not the same unit, and the source does not reconcile the printed label with the stated computation.

metric definition

Appendix D.2.2 'Evaluation metrics': Pearson R, Kendall tau, pairwise MAE (PMAE), MAE and the fraction of predictions within 1 and 2 kcal/mol, each computed per assay and averaged across assays weighted by compound count, with predictions converted to kcal/mol first; the centered variants translate each assay's predictions to the ground-truth mean

control

Training proteins with at least 90% sequence similarity to a validation or test protein were removed, by clustering all affinity-dataset protein sequences with 'mmseqs easy-cluster ... --min-seq-id 0.9 --cov-mode 0 -c 0.01' and dropping any training protein sharing a cluster with a validation or test protein (appendix D.2.1, 'Data leakage control'). Compound-level dependence was checked in Figure 10 and no strong dependence was observed; that figure prints no values.

heldout test

Held-out affinity test set; training proteins sharing an mmseqs 90% identity cluster with a test protein were removed

source locator

Table 11, 'OpenFE subset of the FEP+ benchmark'

limitations

One value per cell with no uncertainty; bootstrap standard errors appear only as error bars in Figure 6, which prints no numbers.; The column headers read 'target avg.' while appendix D.2.2 defines the aggregation as a compound-weighted mean across assays. Assays and targets are not the same unit and the source does not reconcile the two labels.; No per-assay or per-series value is printed, so the table supports statements about the mean over series and not about any one series.; The units of the PMAE and MAE columns are not printed in the table header; appendix D.2.2 states that all predictions are converted to kcal/mol before the metrics are computed.; The 'Perc. within 1' and 'Perc. within 2' headers say percentage, but appendix D.2.2 defines them as the fraction of predictions with absolute error below the threshold and the printed values lie on a 0 to 1 scale.; The results of FEP+ are intended to show the maximum attainable accuracy of current (commercial) FEP simulations by manually adjusting the protocol to the system at hand after observing the error with respect to the experiment, so the FEP+ arm is not a prospective baseline.; The supplement does not state whether the OpenFE and FEP+ rows were re-run here or taken from the cited prior work, so the origin of those evaluations is recorded as unreported.; This subset and the 4-target subset both derive from the same collection and must partly overlap; the two tables are not independent.; One target can carry several assays, so a compound-weighted mean across assays is not a mean over targets even where the two happen to coincide; Ross et al. 2023 Table 1 surveys this collection's own proteins measured in more than one assay.

missing metadata

uncertainty reporting: reason: unreported; note: Table 11 prints no uncertainty

review

method: source-hash-verification; pdf-text-parse; independent-cell-check; rendered-source-visual-check; ai-assisted-source-review; method note: Re-retrieved all three sources and matched each pinned SHA-256. Re-rendered the Boltz-2 supplement with pdftotext -layout (poppler 26.08.0) and matched the text layer's recorded SHA-256. Re-read all 256 cells with a parser written for this review that does not import the extraction script, matching each result to its cell through its own source_locator, and separately read appendix tables 11, 12 and 14 from PNG renderings of PDF pages 29 and 30 at 160 dpi, and Ross Table 3 from the pinned XML markup. Checked printed value, numeric value, locator, metric concept, qualifier, direction, unit, uncertainty and the linked configuration, protocol and dataset, and that every N/A or '-' cell is recorded as missing with a reason.; reviewer: claude; reviewer note: Claude review agent, independent of the extraction agent; no human review claimed; reviewed at: 2026-10-22T13:40:00Z; artifact sha256: be867a5d92add8b223d47d882a4213ad489741f3e4d5081a0901a54370342569; retrieval url: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC12262699/supplementaryFiles

• part of: Public binding free energy benchmark (Ross et al. 2023)
• uses data: OpenFE subset of the public FEP benchmark
• evaluates task: Small-molecule affinity
• subject: Leakage control applied to the FEP+ benchmark subsets
• assessment: BACPI on the FEP+ benchmark OpenFE subset
• assessment: Boltz-2 on the FEP+ benchmark OpenFE subset
• assessment: Boltz-2 iptm on the FEP+ benchmark OpenFE subset
• assessment: FEP+ on the FEP+ benchmark OpenFE subset
• assessment: GAT on the FEP+ benchmark OpenFE subset
• assessment: OpenFE on the FEP+ benchmark OpenFE subset
• assessed by: Rank close analogues of a hit by binding affinity