limitations: TraitGym official metrics source
Limitations of TraitGym official metrics: AUPRC by chromosome, weighted average (Hugging Face dataset songlab/TraitGym)
["Official benchmark repository by the TraitGym authors, who also developed GPN-MSA and GPN-Promoter; those rows are author_reported.", "Benchmark paper: Benegas, Eraslan and Song, bioRxiv 10.1101/2025.02.11.637758 v2 (2025-03-04, CC BY-NC-ND), outside the search window; its Figure 2 plots these metrics. Its Table S8 prints the same zero-shot AUPRCs rounded to three decimal places, on the same chromosome-weighted basis: Appendix B.4 says \"To allow easy comparison, we also use the weighted average AUPRC to evaluate zero-shot scores, even though it is not strictly necessary\", and the global-AUPRC version is reported separately in Supplementary Figure S7. All 100 cells match the values here and were reconciled rather than re-extracted, in data/omics/traitgym-20261116; the paper source is linked with companion_of.", "Hugging Face datasets are mutable; the revision hash pins this retrieval."]
Keeps the previous limitations of the TraitGym metrics source record. The second entry said the paper's Table S8 was on an unweighted basis, which the paper's Appendix B.4 contradicts.
Evidence
Source checking verifies the cited claim or transcription. It does not establish independent reproduction.
Evidence table
Inspect claims, sources and review details
Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.
One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.
7 evidence rows matching the loaded filters
| Property and statement | Original source and location | Review and provenance |
|---|---|---|
| Claimed field attributes.limitations Context-only references | Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2) Appendix B.4 (page 24); Section 'Additional metrics' (page 7); Table S8 (page 27) Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer. |
| Previous value ["Official benchmark repository by the TraitGym authors, who also developed GPN-MSA and GPN-Promoter; those rows are author_reported.", "Benchmark paper: Benegas, Eraslan and Song, bioRxiv 10.1101/2025.02.11.637758 v2 (2025-03-04, CC BY-NC-ND), outside the search window; its Figure 2 plots these metrics. The paper's Table S8 prints rounded zero-shot AUPRCs on a related but unweighted basis and was not extracted.", "Hugging Face datasets are mutable; the revision hash pins this retrieval."] Context-only references | Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2) Appendix B.4 (page 24); Section 'Additional metrics' (page 7); Table S8 (page 27) Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer. |
| Source location Appendix B.4 (page 24); Section 'Additional metrics' (page 7); Table S8 (page 27) Context-only references | Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2) Appendix B.4 (page 24); Section 'Additional metrics' (page 7); Table S8 (page 27) Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer. |
| Claimed value ["Official benchmark repository by the TraitGym authors, who also developed GPN-MSA and GPN-Promoter; those rows are author_reported.", "Benchmark paper: Benegas, Eraslan and Song, bioRxiv 10.1101/2025.02.11.637758 v2 (2025-03-04, CC BY-NC-ND), outside the search window; its Figure 2 plots these metrics. Its Table S8 prints the same zero-shot AUPRCs rounded to three decimal places, on the same chromosome-weighted basis: Appendix B.4 says \"To allow easy comparison, we also use the weighted average AUPRC to evaluate zero-shot scores, even though it is not strictly necessary\", and the global-AUPRC version is reported separately in Supplementary Figure S7. All 100 cells match the values here and were reconciled rather than re-extracted, in data/omics/traitgym-20261116; the paper source is linked with companion_of.", "Hugging Face datasets are mutable; the revision hash pins this retrieval."] Context-only references | Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2) Appendix B.4 (page 24); Section 'Additional metrics' (page 7); Table S8 (page 27) Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer. |
| Description Keeps the previous limitations of the TraitGym metrics source record. The second entry said the paper's Table S8 was on an unweighted basis, which the paper's Appendix B.4 contradicts. Context-only references | Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2) Appendix B.4 (page 24); Section 'Additional metrics' (page 7); Table S8 (page 27) Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer. |
| Relationship: subject lit-genomics-20261011-traitgym-source-hf-metrics Context-only references | Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2) Appendix B.4 (page 24); Section 'Additional metrics' (page 7); Table S8 (page 27) Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer. |
| Name limitations: TraitGym official metrics source Context-only references | Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2) Appendix B.4 (page 24); Section 'Additional metrics' (page 7); Table S8 (page 27) Version: bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material) | not individually reviewed No individual claim review recorded Audit detailsField: Source artifact SHA-256: Hash scope: SHA-256 of the PDF bytes of one download. bioRxiv stamps each download, so this hash pins this retrieval and not the document: a second download in the same session gave 5f424797ed5c92a3852a52e8ecf29f0c3a3416740a47fe9f9fefe9fd9fb0c876. The stable check is the pdftotext -layout text layer, whose SHA-256 is 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097 and was byte-identical across both downloads; every value in this batch was read from that text layer. |
Sources and history
Release 2026-11-21-c8bea25e5abb · Record review: source checked
1 source record and release history
- Benegas, Eraslan and Song 2025, TraitGym benchmark preprint (bioRxiv v2) · Original source · bioRxiv version 2, posted 2025-03-04 (37 pages, main text and Supplementary Material)
Technical metadata and extraction receipts
Stable ID: metadata-correction-traitgym-20261116-hf-metrics-source-limitations
- areas
- dna-genomes
- contexts
- research
- field
- attributes.limitations
- previous value
- ["Official benchmark repository by the TraitGym authors, who also developed GPN-MSA and GPN-Promoter; those rows are author_reported.", "Benchmark paper: Benegas, Eraslan and Song, bioRxiv 10.1101/2025.02.11.637758 v2 (2025-03-04, CC BY-NC-ND), outside the search window; its Figure 2 plots these metrics. The paper's Table S8 prints rounded zero-shot AUPRCs on a related but unweighted basis and was not extracted.", "Hugging Face datasets are mutable; the revision hash pins this retrieval."]
- review
- artifact sha256: 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097; method: source-hash-verification; pdf-text-parse; transcription; scope-review; method note: The pinned artifact was re-downloaded from the v2 full-PDF URL for this pass. Its PDF bytes hash to 961933ed2396ecb0a75737cc27f0655db07ca62be49654f9139ae95815d050f9, not to the 80426c499e0b498e07c07ea598a2f68049adea7569b30fda58cdc0ebafa6c7aa on the source record, and cannot: bioRxiv restamps the container on every request. The reviewer's download differs from the extractor's two in 41 and 44 bytes, every one of them a download-time stamp (the XMP ModifyDate and MetadataDate, the document ModDate and the trailer ID), with the same 37 pages, the same CreationDate of 2025-03-04 and the same pdftotext -layout text layer. That text layer hashes to 23ae4d67751f1092e5cc2da66b52091fa9bbc3ec59310f06bc65be28f93c5097, which is the hash the source record's hash_scope names as the stable check and the hash recorded here; every value in this batch was read from it.; note: The claim that the paper's Table S8 is on "a related but unweighted basis" is wrong. Appendix B.4 ends "To allow easy comparison, we also use the weighted average AUPRC to evaluate zero-shot scores, even though it is not strictly necessary", and the main text puts the global-AUPRC version in Supplementary Figure S7, so the weighted average is the basis of Table S8 and the global AUPRC is the alternative. The reconciliation then holds: all 100 Table S8 cells equal the stored weighted values rounded to three decimal places. The entry mattered because it told a reader the paper's values and the stored values are not comparable when they are the same numbers. The 100-cell reconciliation of Table S8 against the stored Hugging Face metric results was re-done independently by review/reconcile_s8.py, with the cell-to-metric-file mapping written out again rather than reused. Four comparisons, all with no differences: the reviewer's reading of the paper against the stored full-precision values rounded to three decimal places, the reviewer's reading against the extractor's paper values, the reviewer's rounding against the extractor's, and the reviewer's cell-to-stored-result mapping against the extractor's (review/reconcile-s8.tsv). All 156 stored results were accounted for again from their metric file names alone: 100 Table S8 cells, 36 linear probing (.LogisticRegression.chrom, 18 systems on each dataset), 14 zero-shot results of seven non-gLM systems on both datasets, and 6 Mendelian-only duplicate scoring variants, with nothing left over.; retrieval url: https://www.biorxiv.org/content/10.1101/2025.02.11.637758v2.full.pdf; reviewed at: 2026-11-16T02:10:00Z; reviewer: claude; reviewer note: Claude review agent (independent of the extractor); no human review claimed. Every value was read against the pinned artifact; nothing was reproduced and no model was run.
- source locator
- Appendix B.4 (page 24); Section 'Additional metrics' (page 7); Table S8 (page 27)
- value
- ["Official benchmark repository by the TraitGym authors, who also developed GPN-MSA and GPN-Promoter; those rows are author_reported.", "Benchmark paper: Benegas, Eraslan and Song, bioRxiv 10.1101/2025.02.11.637758 v2 (2025-03-04, CC BY-NC-ND), outside the search window; its Figure 2 plots these metrics. Its Table S8 prints the same zero-shot AUPRCs rounded to three decimal places, on the same chromosome-weighted basis: Appendix B.4 says \"To allow easy comparison, we also use the weighted average AUPRC to evaluate zero-shot scores, even though it is not strictly necessary\", and the global-AUPRC version is reported separately in Supplementary Figure S7. All 100 cells match the values here and were reconciled rather than re-extracted, in data/omics/traitgym-20261116; the paper source is linked with companion_of.", "Hugging Face datasets are mutable; the revision hash pins this retrieval."]