Huang, Knight and Naprienko 2025, arXiv:2507.21404v2 (abstract page)

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 1 benchmark
• 4 evidence claims
• 1 use case

Claims

Statements this source supports, each with where in the source it was checked.

• LIT-PCBA leakage audit source: evidence_concerns removed: Evidence concerns Removed. The source record now carries no evidence_concerns entry. The audit's finding is kept as the claim hitdisc-20261022-claim-litpcba-leakage-audit, as limitations on the LIT-PCBA benchmark record and the five-target protocol, and as a source warning on each of the 84 LIT-PCBA results. (arXiv:2507.21404v2 abstract page; shared/omics/use-cases.ts cleanSource)
• LIT-PCBA leakage audit: a parameter-free memorisation baseline matches the state of the art: Reported finding Huang, Knight and Naprienko report extensive data leakage and molecular redundancy across the LIT-PCBA splits: 2D-identical ligands within and across partitions, pervasive analogue overlap and low-diversity query sets. In ALDH1 alone 323 active training-validation analogue pairs occur at ECFP4 Tanimoto at or above 0.6; across all targets 2,491 2D-identical inactives appear in both training and validation, with very few corresponding actives. They state that these overlaps let models succeed through scaffold memorisation rather than generalisation, inflating enrichment factors and AUROC scores, and that the flaws are severe enough that a trivial memorisation-based baseline with no learnable parameters can match or exceed the reported performance of state-of-the-art deep-learning and 3D-similarity models. They conclude that nearly all published results on LIT-PCBA are undermined, that even models evaluated in zero-shot mode are affected by analogue leakage into the query set, and that the benchmark in its current form does not measure a model's ability to recover novel chemotypes. (Abstract)
• Relevance of hitdisc-20261022-protocol-shen-litpcba5 to use-case-hit-discovery-virtual-screening: Assessed by LIT-PCBA five-target screening (Shen et al. 2025 Table 4) (hitdisc-20261022-protocol-shen-litpcba5: see its source_locator)
• Summary of the evidence for use-case-hit-discovery-virtual-screening: Summary Three sources in this store score machine-learning and co-folding methods against docking on the same screening data, and what the numbers say depends on how the negatives were built. On DEKOIS 2.0, whose 79 targets each pair 40 validated actives with 1,200 property-matched decoys selected from ZINC (1:30), the co-folding confidence scores enrich strongly and beat the docking arms. AF3 min-iPAE gives a mean EF1% of 23.15 and Boltz-2 min-iPAE 22.73, against 12.01 for the Glide SP row quoted from another paper, 5.16 for Gnina Vina and 2.94 for TankBind; the best rescoring of a co-folded pose, RTMScore on AF3 poses, reaches 21.34. The authors flag their own Boltz-2 affinity arms (mean EF1% 23.54 and 25.84) as likely over-estimates because Boltz-2 drew affinity data from the same public databases that source DEKOIS 2.0. On GPCRrecent, 16 targets whose first crystal structures were released after 2022 with decoys generated by a conditional recurrent neural network at 1:50, every value falls and the margin narrows: Boltz-2 min-iPAE gives a mean EF1% of 13.96 against 5.02 for Glide SP and 3.65 for Gnina Vina. On the five LIT-PCBA targets where the inactives were experimentally measured, the margin largely disappears: averaged over the five targets Boltz-2 min-iPAE gives EF1% 9.30 and NEF1% 0.100 against Glide SP's 6.79 and 0.079, and on TP53 both Boltz-2 scores give EF1% 0.00 while Protenix ipTM gives 5.00 and Glide SP 3.75. A separate audit of LIT-PCBA reports 323 active training-validation analogue pairs at ECFP4 Tanimoto at or above 0.6 in ALDH1 alone and 2,491 2D-identical inactives shared between training and validation across targets, and concludes that a parameter-free memorisation baseline matches or exceeds published deep-learning results there, so every LIT-PCBA value above carries that limit. Where the negatives are compounds a docking screen already ranked highly and every compound was synthesised and assayed, the ordering changes. On the AmpC list (247 inhibitors among 1,293 tested compounds from a 1.7-billion molecule screen) DOCK3.7 separates binders best, AUC 76.5, against 68.1 for Boltz-2 pIC50 and 60.5 for AF3 L-pLDDT. On the dopamine D4 list (205 binders among 541) docking and Boltz-2 are indistinguishable at the precision printed (both about 71) and AF3 L-pLDDT is worse than random at 46.4. Only on the sigma-2 list (201 binders among 506) does a co-folding score lead, Boltz-2 pIC50 at 83.8 against DOCK3.7's 78.8, with AF3 L-pLDDT at 56.1. In the one blinded prospective challenge, CACHE #3 on the SARS-CoV-2 Nsp3 macrodomain, 23 workflows selected 1,739 compounds that were synthesised and assayed; 28 advanced to hit expansion and 29 confirmed hits scored above 15, four of them judged chemically novel. Twelve of the 23 workflows returned no confirmed hit from their own selection of 41 to 99 compounds. When every workflow was applied to the same merged 1,739-compound library, WF1696, which the article describes as a purely physics-based method, captured 2 of the 4 Tier 1 and 9 of the 25 Tier 2 confirmed hits from 99 predicted actives, which the authors report as did better than any other method in retrieving hits from the collection of all molecules selected in Round 1. Cost bounds all of this: co-folding one protein-ligand pair takes 7.79 to 34.58 seconds per compound on one NVIDIA H100 with pre-computed MSAs, which is why the LIT-PCBA arm covers five targets rather than fifteen. No docking tool was timed on the same hardware, no ligand-similarity or pharmacophore baseline is scored anywhere in this batch, and no result here estimates the hit rate of an untested library. (Shen et al. 2025 Tables 1-5; Kim, Correy, Hall et al. 2026 Results; Herasymenko et al. 2026 Results and sheet S8; arXiv:2507.21404v2 Abstract)

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.