LIT-PCBA leakage audit: a parameter-free memorisation baseline matches the state of the art
Reported finding of LIT-PCBA
Huang, Knight and Naprienko report extensive data leakage and molecular redundancy across the LIT-PCBA splits: 2D-identical ligands within and across partitions, pervasive analogue overlap and low-diversity query sets. In ALDH1 alone 323 active training-validation analogue pairs occur at ECFP4 Tanimoto at or above 0.6; across all targets 2,491 2D-identical inactives appear in both training and validation, with very few corresponding actives. They state that these overlaps let models succeed through scaffold memorisation rather than generalisation, inflating enrichment factors and AUROC scores, and that the flaws are severe enough that a trivial memorisation-based baseline with no learnable parameters can match or exceed the reported performance of state-of-the-art deep-learning and 3D-similarity models. They conclude that nearly all published results on LIT-PCBA are undermined, that even models evaluated in zero-shot mode are affected by analogue leakage into the query set, and that the benchmark in its current form does not measure a model's ability to recover novel chemotypes.
The headline finding of the LIT-PCBA data-leakage audit, which bears on every LIT-PCBA result.
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
6 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Claimed field reported_finding Context-only references | Huang, Knight and Naprienko 2025, arXiv:2507.21404v2 (abstract page) Abstract Version: arXiv:2507.21404v2, submitted 2025-07-29, this version 2025-08-07 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: This is a mutable web page: artifact_sha256 describes the single retrieval of 2026-10-22T11:59:20Z and a later change to the page would change it. |
| Source location Abstract Context-only references | Huang, Knight and Naprienko 2025, arXiv:2507.21404v2 (abstract page) Abstract Version: arXiv:2507.21404v2, submitted 2025-07-29, this version 2025-08-07 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: This is a mutable web page: artifact_sha256 describes the single retrieval of 2026-10-22T11:59:20Z and a later change to the page would change it. |
| Claimed value Huang, Knight and Naprienko report extensive data leakage and molecular redundancy across the LIT-PCBA splits: 2D-identical ligands within and across partitions, pervasive analogue overlap and low-diversity query sets. In ALDH1 alone 323 active training-validation analogue pairs occur at ECFP4 Tanimoto at or above 0.6; across all targets 2,491 2D-identical inactives appear in both training and validation, with very few corresponding actives. They state that these overlaps let models succeed through scaffold memorisation rather than generalisation, inflating enrichment factors and AUROC scores, and that the flaws are severe enough that a trivial memorisation-based baseline with no learnable parameters can match or exceed the reported performance of state-of-the-art deep-learning and 3D-similarity models. They conclude that nearly all published results on LIT-PCBA are undermined, that even models evaluated in zero-shot mode are affected by analogue leakage into the query set, and that the benchmark in its current form does not measure a model's ability to recover novel chemotypes. Context-only references | Huang, Knight and Naprienko 2025, arXiv:2507.21404v2 (abstract page) Abstract Version: arXiv:2507.21404v2, submitted 2025-07-29, this version 2025-08-07 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: This is a mutable web page: artifact_sha256 describes the single retrieval of 2026-10-22T11:59:20Z and a later change to the page would change it. |
| Description The headline finding of the LIT-PCBA data-leakage audit, which bears on every LIT-PCBA result. Context-only references | Huang, Knight and Naprienko 2025, arXiv:2507.21404v2 (abstract page) Abstract Version: arXiv:2507.21404v2, submitted 2025-07-29, this version 2025-08-07 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: This is a mutable web page: artifact_sha256 describes the single retrieval of 2026-10-22T11:59:20Z and a later change to the page would change it. |
| Relationship: subject hitdisc-20261022-benchmark-litpcba Context-only references | Huang, Knight and Naprienko 2025, arXiv:2507.21404v2 (abstract page) Abstract Version: arXiv:2507.21404v2, submitted 2025-07-29, this version 2025-08-07 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: This is a mutable web page: artifact_sha256 describes the single retrieval of 2026-10-22T11:59:20Z and a later change to the page would change it. |
| Name LIT-PCBA leakage audit: a parameter-free memorisation baseline matches the state of the art Context-only references | Huang, Knight and Naprienko 2025, arXiv:2507.21404v2 (abstract page) Abstract Version: arXiv:2507.21404v2, submitted 2025-07-29, this version 2025-08-07 | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: This is a mutable web page: artifact_sha256 describes the single retrieval of 2026-10-22T11:59:20Z and a later change to the page would change it. |
Sources and history
Release 2026-10-22-1b7cf027e4f6 · Record review: source checked
1 source record and release history
- Huang, Knight and Naprienko 2025, arXiv:2507.21404v2 (abstract page) · Original source · arXiv:2507.21404v2, submitted 2025-07-29, this version 2025-08-07
Technical metadata and extraction receipts
Stable ID: hitdisc-20261022-claim-litpcba-leakage-audit
- areas
- molecular-interactions
- contexts
- research
- field
- reported_finding
- value
- Huang, Knight and Naprienko report extensive data leakage and molecular redundancy across the LIT-PCBA splits: 2D-identical ligands within and across partitions, pervasive analogue overlap and low-diversity query sets. In ALDH1 alone 323 active training-validation analogue pairs occur at ECFP4 Tanimoto at or above 0.6; across all targets 2,491 2D-identical inactives appear in both training and validation, with very few corresponding actives. They state that these overlaps let models succeed through scaffold memorisation rather than generalisation, inflating enrichment factors and AUROC scores, and that the flaws are severe enough that a trivial memorisation-based baseline with no learnable parameters can match or exceed the reported performance of state-of-the-art deep-learning and 3D-similarity models. They conclude that nearly all published results on LIT-PCBA are undermined, that even models evaluated in zero-shot mode are affected by analogue leakage into the query set, and that the benchmark in its current form does not measure a model's ability to recover novel chemotypes.
- source locator
- Abstract
- review
- method: scope-review; automated-source-review; reviewer: claude; reviewer note: Claude review agent, independent of the extraction agent; no human review claimed; date: 2026-10-22; method note: Read the record against the pinned sources and against docs/use-cases.md and docs/record-contract.md.; note: Claim checked against the pinned source. Every figure it states was matched to a result record in this batch.; artifact sha256: e766c8aa4f94287672b2b1bacab2480a41ec7c806ecbaf5e2d1bf8773781310f; retrieval url: https://arxiv.org/abs/2507.21404v2; receipt: data/omics/use-case-coverage-hitdisc-20261022/review.json
Related records
- subject: LIT-PCBA