rewirebio.iobenchmarks

Use caseMetabolomicsResearch only

Shortlist molecular identities from tandem mass spectra

Which retrieval configurations merit validation when molecular formula is unknown and a researcher can examine only a short list of candidate structures?

What the evidence shows

Two preprints report formula-free retrieval of the true structure from mass-matched PubChem candidates for MassSpecGym spectra, and their values are not directly comparable.

In the MSAlign preprint, MSAlign ranks first among the formula-free methods on every split the paper reports, but its top-1 recall falls from about half on the formula split to under a fifth on the MCES split. On the MassSpecGym formula split, Recall@1 was 47.5 for MSAlign and 58.2 with score fusion, against 42.7 for Emb-Cos, 13.9 for JESTR and 4.7 for DeepSet, with standard deviations of 0.5 to 3.4 over three random splitting seeds. On the MCES split the order is the same but the values are much lower: 19.0, 30.8, 12.6, 7.9 and 1.9; the table caption describes three splits there too, but the methods describe two swapped variants.

Recall@20 reaches 87.0 for MSAlign and 92.8 with score fusion on the formula split, and 64.9 and 82.0 on MCES. An earlier version of the same paper reports Recall@1 of 53.8 for MSAlign and 41.2 for Emb-Cos on the formula split, so the two versions differ and are not independent measurements. These MSAlign values assume the true structure is among 256 mass-matched candidates and were scored by the MSAlign authors.

MassSpecGym in the Wild, whose authors include the developers of MIST, MIST-CF and MassSpecGym, reports two formula-free configurations that it does not flag for formula leakage, on the MassSpecGym test fold with up to 256 mass-matched candidates per spectrum. MIST given the top formula predicted by MIST-CF found the true structure at rank 1 for 26.64 percent of test spectra and within the top 20 for 54.90 percent, against 0.43 and 7.99 percent for random ranking. Its candidate pool and split differ from MSAlign's, and the MIST score depends on the formula annotator as well as the retriever.

All of these values are reported by the authors of preprints, and none measures open-world identification, because the true structure is always among the candidates.

You bring
  • MS/MS spectra with metadata sufficient to derive neutral molecular mass
  • A frozen set of candidate molecular structures; the true identity must be present for the measured retrieval endpoint
You want
Split-specific Recall@1 and Recall@20 evidence for six exact configurations, plus candidate-pool and domain-shift limits.
Evidence in this release
8 comparisons shown · 9 tools · 13 configurations
Kept up to date
Evidence last checked 7 October 2026 · re-checked monthly, next due 7 November 2026
Executive Takeaway

For "Shortlist molecular identities from tandem mass spectra", on MSAlign molecular retrieval MassSpecGym formula split: no formula, R@1: MassSpecGym formula split retrieval with no formula, Top-1 accuracy (retrieval) (Top-1 accuracy, a proxy protocol), MSAlign scored 53.8%, reported by msalign: Primary paper PDF.

Best reported valueMSAlignmsalign: Primary paper PDF
Top-1 accuracy53.8%MSAlign molecular retrieval MassSpecGym formula split: no formula, R@1: MassSpecGym formula split retrieval with no formula, Top-1 accuracy (retrieval)
Baseline on this protocolNone scoredNo margin is computed
Ranked tools65 from independent studies

6 tools have a value on this protocol; 5 of them come from studies whose authors did not develop the tool. No baseline method is scored on this protocol. Reviewers judged this protocol a proxy for the use case.

Ranking

Ranked on MSAlign molecular retrieval MassSpecGym formula split: no formula, R@1: MassSpecGym formula split retrieval with no formula, Top-1 accuracy (retrieval) (Top-1 accuracy, higher is better). Proxy protocol Reviewers judged this protocol a proxy for the use case, not a direct test of it. Only results on this protocol are ranked; other tools are listed below the table.

6 tools, from 1 source. 5 of 6 results (83%) come from an independent study; 1 is reported by the tool's developers. The top-ranked result is reported by the tool's developers.

Rank▲ModelTop-1 accuracy (higher is better)EvidenceCoverageReported byLicense
#153.8%Source-checked Developer-reported7/8msalign: Primary paper PDFNot recorded
#241.2%Source-checked Independent study4/8msalign: Primary paper PDFNot recorded
#329.5%Source-checked Independent study4/8msalign: Primary paper PDFNot recorded
#422.2%Source-checked Independent study4/8msalign: Primary paper PDFNot recorded
#520.0%Source-checked Independent study4/8msalign: Primary paper PDFNot recorded
#616.9%Source-checked Independent study4/8msalign: Primary paper PDFNot recorded
5 tools are evaluated elsewhere in this use case but not on this protocol, so not ranked

Baseline

MassSpecGym DeepSets fingerprint baseline. No pre-machine-learning retrieval method, such as spectral library search, is measured in the linked evidence, so the reference is the benchmark's own DeepSets fingerprint predictor: a small trained network that ranks candidates by predicted-fingerprint similarity. The only non-learned rows are random rankings of the candidate list, a chance-level control rather than a method to beat.

Machine-learning methods are compared with the baseline on the same data in 3 tables: MSAlign v2 retrieval: massspecgym-mces-formula-free, MSAlign v2 retrieval: massspecgym-formula-formula-free, MSAlign v2 retrieval: spectraverse-formula-free. Baseline rows are marked in each table.

Tools compared

Comparisons

Unified model comparison for this use case. Compare tools against protocol baselines in a single view.

MSAlign molecular retrieval MassSpecGym formula split: no formula, R@1: MassSpecGym formula split retrieval with no formula

Proxy evidenceDeveloper-reported: Reported by the developers of the evaluated tool.Independent study: Reported in a study whose authors did not develop the evaluated tool.

Recall@1 of the true molecular structure among 256 mass-matched candidate structures on the MassSpecGym formula split, with no molecular formula supplied.

Sorted by Top-1 accuracy (higher is better). The best value in each column is highlighted.

ToolTop-1 accuracy (retrieval)
MSAlignfoundation model
53.8% (best in column)
Emb-Cosmachine learning
41.2%
SAILmachine learning
29.5%
DeepSetsmachine learning
22.2%
JESTRmachine learning
20.0%
FFNmachine learning
16.9%
Caveats and method
  • The correct structure must be in a 256-candidate mass-filtered pool. Scores do not transfer automatically to a larger database, a different pool-construction procedure or a missing true structure.
  • MCES and formula splits impose different distribution shifts. The source authors favour formula splits for their deployment interpretation; that choice does not prove that formula-split performance represents every intended application.
  • Reported methods are the paper implementations: DeepSets training was extended to 50 epochs. These are not interchangeable with original model-paper scores or current checkpoints.
  • Exact test denominators, split-manifest hashes, checkpoint hashes and evaluator revisions remain unextracted or unreported in the catalogue. No uncertainty values or new independent reproduction are established.
  • Candidate recall is not an authenticated metabolite identification, a clinical diagnosis or a calibrated probability. Human domain review remains outstanding.

Protocol: MSAlign molecular retrieval MassSpecGym formula split: no formula, R@1: MassSpecGym formula split retrieval with no formula

Primary benchmark evaluation shown (MSAlign molecular retrieval MassSpecGym formula split: no formula, R@1: MassSpecGym formula split retrieval with no formula). 7 additional study protocols (MSAlign v2 retrieval: massspecgym-mces-formula-free, MSAlign v2 retrieval: massspecgym-formula-formula-free, MSAlign v2 retrieval: spectraverse-formula-free and others) are cited in individual tool drawer rows and under Question, sources and review.

Not covered yet

  • The correct structure must be in a 256-candidate mass-filtered pool. Scores do not transfer automatically to a larger database, a different pool-construction procedure or a missing true structure.
  • MCES and formula splits impose different distribution shifts. The source authors favour formula splits for their deployment interpretation; that choice does not prove that formula-split performance represents every intended application.
  • Reported methods are the paper implementations: DeepSets training was extended to 50 epochs. These are not interchangeable with original model-paper scores or current checkpoints.
  • Exact test denominators, split-manifest hashes, checkpoint hashes and evaluator revisions remain unextracted or unreported in the catalogue. No uncertainty values or new independent reproduction are established.
  • Candidate recall is not an authenticated metabolite identification, a clinical diagnosis or a calibrated probability. Human domain review remains outstanding.
  • The four metric and split groups come from one paper and do not constitute independent replications.
  • 30 September 2026 audit: v1 source record labels 2605.19752v1 but uses an unversioned URL that now serves v2; old source bytes/values must remain immutable. v1-specific URL returned HTTP406 during this audit.
  • 30 September 2026 audit: For MCES, the v2 Table3 generic caption says three random splits, but Section5.1 explicitly uses two validation/test-swapped variants; the discrepancy is preserved.
  • 30 September 2026 audit: Benchmark assumes the true structure is present among 256 neutral-mass-matched candidates. Unknown candidates, instrument-specific prospective performance and authenticated identification remain gaps.

Contribute evidence or propose a correction

How to cite

Shortlist molecular identities from tandem mass spectra. Rewire biological model benchmark database, release 2026-11-21-c8bea25e5abb. https://benchmarks.rewirebio.io/use-cases/mass-spectrum-molecule-shortlisting/

Release 2026-11-21-c8bea25e5abb. Rankings and values change between releases, so keep the release id in the citation. Cite the original publication as well for any value you reuse: each result links its source.

The compilation and annotations made by Rewire in this release are offered under CC BY 4.0; values and text taken from cited sources stay under the terms of those sources and are not covered by this licence. Data licence

Question, sources and review

Who this is for, setting and exclusions

Compare formula-free retrieval configurations within a named split and shortlist size, then validate the intended instrument, chemistry and candidate database before selecting a workflow.

Who this is for
  • Metabolomics researchers shortlisting candidate molecular identities for experimental confirmation
  • Computational researchers choosing retrieval inputs, candidate pools and validation splits
Biological setting
MSAlign paper Table 3, MassSpecGym formula and MCES splits, formula-free inputs. Each source candidate pool contains 256 mass-matched PubChem structures. Formula and MCES splits remain distinct.
Research setting
Research only. Candidate retrieval is not a confirmed molecular identification or evidence of clinical diagnostic performance.

Outside this use case

  • De novo molecular generation or cases where the true structure is absent from the candidate set
  • Formula-conditioned retrieval, including MIST, FLARE and MSAlign+Filter
  • Pooling split-specific results or transferring scores to different candidate database sizes
  • Authenticated molecular identification and clinical diagnosis
Sources, review and downloads

Automated source review · 2026-09-30 · Codex source curation, independent worker cross-review and root integration review

Source-backed literature curation with independent automated transcription and scope checks. No new model execution, independent experimental replication, qualified human scientific review or clinical validation.

Release 2026-11-21-c8bea25e5abb · question use-case-mass-spectrum-molecule-shortlisting

Use-case input digest bfbb7ab9303becad1e3cb395568c26f41e72ddb67cd1a89a06d9eaa0f76bd95f

Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)