MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery

What this source contributed

Records in this release that cite this source. A record can cite several sources.

• 12 models
• 4 methods
• 34 configurations
• 14 protocols
• 3 datasets
• 156 evaluations
• 156 results
• 3 evidence claims

Results

Values as the source printed them. Compare results only under compatible protocols, datasets and metrics.

Claims

Statements this source supports, each with where in the source it was checked.

• Evidence summary: Shortlist molecular identities from tandem mass spectra: Summary Two preprints report formula-free retrieval of the true structure from mass-matched PubChem candidates for MassSpecGym spectra, and their values are not directly comparable. In the MSAlign preprint, MSAlign ranks first among the formula-free methods on every split the paper reports, but its top-1 recall falls from about half on the formula split to under a fifth on the MCES split. On the MassSpecGym formula split, Recall@1 was 47.5 for MSAlign and 58.2 with score fusion, against 42.7 for Emb-Cos, 13.9 for JESTR and 4.7 for DeepSet, with standard deviations of 0.5 to 3.4 over three random splitting seeds. On the MCES split the order is the same but the values are much lower: 19.0, 30.8, 12.6, 7.9 and 1.9; the table caption describes three splits there too, but the methods describe two swapped variants. Recall@20 reaches 87.0 for MSAlign and 92.8 with score fusion on the formula split, and 64.9 and 82.0 on MCES. An earlier version of the same paper reports Recall@1 of 53.8 for MSAlign and 41.2 for Emb-Cos on the formula split, so the two versions differ and are not independent measurements. These MSAlign values assume the true structure is among 256 mass-matched candidates and were scored by the MSAlign authors. MassSpecGym in the Wild, whose authors include the developers of MIST, MIST-CF and MassSpecGym, reports two formula-free configurations that it does not flag for formula leakage, on the MassSpecGym test fold with up to 256 mass-matched candidates per spectrum. MIST given the top formula predicted by MIST-CF found the true structure at rank 1 for 26.64 percent of test spectra and within the top 20 for 54.90 percent, against 0.43 and 7.99 percent for random ranking. Its candidate pool and split differ from MSAlign's, and the MIST score depends on the formula annotator as well as the retriever. All of these values are reported by the authors of preprints, and none measures open-world identification, because the true structure is always among the candidates. (uc20260930-source-msalign-v2: Table 3 p7, massspecgym-formula, massspecgym-mces and spectraverse blocks, R@1 and R@20 rows; evidence-official-760ab2fa8c396aeb796c: Table 3, no-formula blocks, R@1 and R@20 rows; lit-otheromics-20261011-msgwild-source-liu2026: Table 2, rows 'Random' and 'MIST (MIST-CF top-1 formula)', columns 'Hit rate@1' and 'Hit rate@20'; Section 4.3)
• Relevance of MassSpecGym mass-based retrieval, Hit rate@1 (MassSpecGym in the Wild Table 2) to "Shortlist molecular identities from tandem mass spectra": Assessed by MassSpecGym mass-based retrieval, Hit rate@1 (MassSpecGym in the Wild Table 2) (lit-otheromics-20261011-msgwild-source-liu2026: Table 2, column 'Hit rate@1')
• Relevance of MassSpecGym mass-based retrieval, Hit rate@20 (MassSpecGym in the Wild Table 2) to "Shortlist molecular identities from tandem mass spectra": Assessed by MassSpecGym mass-based retrieval, Hit rate@20 (MassSpecGym in the Wild Table 2) (lit-otheromics-20261011-msgwild-source-liu2026: Table 2, column 'Hit rate@20')

Models, benchmarks and other records

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.