Rank rare-disease variants for review

• No prospective diagnostic accuracy or equal analyst-time comparison.
• Exact pipeline snapshots and independent clinical review remain outstanding.

Protocol: Talos diagnosis recovery and workload: RGP singleton

• No prospective diagnostic accuracy or equal analyst-time comparison.
• Exact pipeline snapshots and independent clinical review remain outstanding.

Protocol: Talos diagnosis recovery and workload: RGP full

• No prospective diagnostic accuracy or equal analyst-time comparison.
• Exact pipeline snapshots and independent clinical review remain outstanding.

Protocol: Talos diagnosis recovery and workload: RGP trio

• No prospective diagnostic accuracy or equal analyst-time comparison.
• Exact pipeline snapshots and independent clinical review remain outstanding.

Protocol: Talos diagnosis recovery and workload: ACG singleton

• No prospective diagnostic accuracy or equal analyst-time comparison.
• Exact pipeline snapshots and independent clinical review remain outstanding.

Protocol: Talos diagnosis recovery and workload: ACG full

• No prospective diagnostic accuracy or equal analyst-time comparison.
• Exact pipeline snapshots and independent clinical review remain outstanding.

Protocol: Talos diagnosis recovery and workload: ACG trio

• No disease-specific (e.g. BRCA) subgroup claim is supported by this table.
• Short-window and long-window populations are separate and must not be pooled; each configuration/dataset records its exact assignment.
• Sei, Enformer (hidden/output tracks) are non-DNA-foundation specialised genomic comparators per the source's own asterisk notation, not DNA foundation models.
• Supplementary Data 4 gives each configuration's AUC on each of the three held-out chromosome groups. These are stored as separate per-group evaluations and excluded here, so each configuration is shown once, at the three-group mean the paper reports. The three group AUCs differ by 0.0014 to 0.057 within a configuration, and NT-v2 has the highest AUC in every group (0.7312, 0.7318, 0.7326).
• No independent reproduction or qualified human scientific review.

Protocol: Feng pathogenic-versus-common SNP classification

What each column measures:

• GPCards: Hits@1, Hits@10, ROC AUC and AUPR for GPT-4 (zero-shot and one-shot) in GPCards candidate sets of 5 to 100 genes
• ClinVar: Hits@1, Hits@10, ROC AUC and AUPR for GPT-4 (zero-shot and one-shot) and Exomiser 12.1.0 gene scores in ClinVar candidate sets of 5 to 100 genes
• PAVS: Hits@1, Hits@10, ROC AUC and AUPR for GPT-4 (zero-shot and one-shot) and Exomiser 12.1.0 gene scores in PAVS candidate sets of 5 to 100 genes

• Synthetic candidate sets: the causative gene plus randomly chosen genes, not the filtered variant list of a real exome.
• ClinVar phenotypes are OMIM disease annotations from the HPO database, which Exomiser also uses, so the comparison may favour Exomiser (authors' note).
• Phenotypes are database annotations, not observed in a patient workup (PAVS is closer to clinical reports).
• Exomiser was given one random variant per gene and its variant scores were ignored; only gene scores were compared.
• The GPT-4 prompts were designed by the authors, so the GPT-4 rows are author_reported; Exomiser was not developed by them. Two authors (P. N. Schofield, R. Hoehndorf) developed DeepPVP, a competing tool that is not in Table 3.
• Which zero-shot prompt (Q1, Q2, Q3 or Q2Q3) produced the zero-shot columns is not stated; on GPCards with GPT-3.5 the prompts ranged from 30% to 80% Hits@1 (Table 2).
• The prompts were chosen on GPCards, so these values are in-sample for prompt choice, and there is no comparator.
• The ClinVar bias toward Exomiser applies to the ClinVar Exomiser row: its input phenotypes are the HPO annotations of the causal disease in the database Exomiser uses.

Protocols: Ranking the causative gene within synthetic candidate sets of 5 to 100 genes, GPCards (Kafkas et al. 2025 Table 3), Ranking the causative gene within synthetic candidate sets of 5 to 100 genes, ClinVar (Kafkas et al. 2025 Table 3), Ranking the causative gene within synthetic candidate sets of 5 to 100 genes, PAVS (Kafkas et al. 2025 Table 3)

What each column measures:

• DDD (305): Top-1 to top-50 causal-gene recovery for 11 tool configurations (6 using HPO and VCF, 5 HPO-only) on 305 solved DDD exomes, default singleton runs
• KMCGD (209): The same metrics on 209 solved in-house exomes

• Ranking of the known causal gene in solved cases; does not establish pathogenicity or a diagnosis.
• Each case has a single causal gene; cases with no or several causal genes are not represented.
• Default parameters and singleton mode for every tool; versions from 2014 to 2020.
• AMELIE draws on the literature, and 156 of 305 DDD cases are published (HGMD); the authors report an almost unchanged trend on the 149 unpublished cases.
• Same group later optimised parameters (Yuan et al. 2024); the two studies use different tool versions.
• Five configurations (Phenolyzer, HANRD, GADO, Phen2Gene, AMELIE-HPO) rank genes from HPO terms alone without the variant calls; their comparison fields keep them apart from the HPO plus VCF runs.
• Only solved cases with a single causal gene (a single SNV or indel rated definitely pathogenic in DDD) are scored; unsolved cases, cases with several causal variants and cases outside the tools' variant classes are not represented, so recovery rates are higher than in an unselected cohort.
• Run by the KingMed laboratory group; no author developed a compared tool (checked against the author lists of the tools' cited papers).
• Only solved cases with a single causal gene (Sanger-confirmed, ACMG/AMP) are scored; unsolved cases, cases with several causal variants and cases outside the tools' variant classes are not represented, so recovery rates are higher than in an unselected cohort.
• Mixed clinical-genetics population with sparse phenotypes (mean 2.0 HPO terms); not restricted to congenital anomalies or developmental disorders.
• In-house cohort from the authors' laboratory; not public.
• Default parameters and singleton mode for every tool.

Protocols: Causal-gene rank in 305 DDD exomes, default singleton runs (Yuan et al. 2022 SM Table 3), Causal-gene rank in 209 in-house exomes, default singleton runs (Yuan et al. 2022 SM Table 3)

What each column measures:

• DDD trios (305): Top-1 to top-50 causal-gene recovery for Exomiser 13.1.0, PhenIX 1.16 and AMELIE 3.1.0 under four parameter protocols each and LIRICAL 1.3.4, on 305 DDD trio families
• KGD trios (152): The same metrics on 152 in-house trio families

• Ranking of the known causal gene in solved cases; does not establish pathogenicity or a diagnosis.
• Each case has a single causal gene; cases with no or several causal genes are not represented.
• Only Exomiser and PhenIX Protocols C and D are trio mode. Every other row, including all AMELIE and LIRICAL rows, is singleton (Discussion paragraph 1, item 4). The authors' comparison of optimised settings sets trio-mode Exomiser and PhenIX (Protocol D) against singleton AMELIE (Protocol B) and LIRICAL; compare within mode only.
• The authors chose Protocol D for Exomiser and PhenIX and Protocol B for AMELIE on these same cohorts, so those six rows (two cohorts) overstate expected performance on new cases. The values of all rows are as printed.
• Run by the KingMed laboratory group; no author developed a compared tool (checked against the author lists of the tools' cited papers).
• What distinguishes Exomiser and PhenIX Protocol C from Protocol D is shown only in the Figure 2 option table (image, not read); the text says only that both are trio mode. The pathogenicity source for C and D is not stated in the text.
• Only solved cases with a single causal gene (a single SNV or indel rated definitely pathogenic in DDD) are scored; unsolved cases, cases with several causal variants and cases outside the tools' variant classes are not represented, so recovery rates are higher than in an unselected cohort.
• In-house cohort; not public.
• 58 of the 152 KGD families are carried over from the Yuan et al. 2022 in-house cohort.

Protocols: Causal-gene rank in 305 DDD trio-family exomes across parameter protocols (Yuan et al. 2024 Table 1), Causal-gene rank in 152 KGD trio-family exomes across parameter protocols (Yuan et al. 2024 Table 1)

What each column measures:

• DMS benchmark, all 36 DMS sets: Average pairwise win rate of each missense VEP against 96 others, by |Spearman| with 36 human DMS datasets
• DMS benchmark, direct assays only: Average pairwise win rate by |Spearman| with the DMS assays the authors class as direct
• DMS benchmark, indirect assays only: Average pairwise win rate by |Spearman| with the DMS assays the authors class as indirect
• DMS benchmark, SNV-accessible substitutions: Average pairwise win rate by |Spearman| restricted to substitutions reachable by one nucleotide change
• DMS benchmark, SNV-accessible, gaps filled: Average pairwise win rate by |Spearman| on SNV-accessible substitutions with missing predictions set to each VEP's most benign score
• Clinical benchmark, ClinVar P/LP versus gnomAD missense: Average pairwise win rate by per-protein AUROC separating ClinVar P/LP from gnomAD v4.1 missense variants in 819 proteins

• DMS phenotypes are chosen per protein and are not pathogenicity labels.
• Rank scores are relative to the VEP set compared; they are not a correlation and cannot be compared with other tables.
• Correlation is assessed within each protein, so the ranking does not test whether scores are comparable across genes; the authors report that DMS and clinical rankings agree closely for population-free and population-tuned VEPs (Pearson r 0.978 and 0.994) and less for clinical-trained VEPs (Results, Fig. 4).
• Transcribed from the source; no reproduction by Rewire and no human scientific review.
• Which assays are direct is not fully recoverable from Table S1 (four blank assay descriptions).
• Which assays are indirect is not fully recoverable from Table S1.
• The negative class is gnomAD v4.1 missense variants not classed pathogenic ('putatively benign'); the authors note it will include unclassified pathogenic variants, particularly in recessive and incomplete-penetrance genes (Results).
• VEPs that use allele frequency as a feature likely benefit from circularity, because frequency is used to classify variants as benign (Discussion); the gnomAD negative class reduces but does not remove this.
• No training variants were removed; the authors attribute most of the clinical-trained VEPs' advantage over the DMS ranking to circularity (Results, Fig. 4).
• AUROC is computed per protein, so the ranking does not test whether scores are comparable across genes.
• The text says 83 VEPs; Table S8 prints 85 rows (see the protocol's limitations).
• The use case excludes balanced pathogenic/benign classification as a measure of case-level diagnostic performance; this protocol is variant-level classification and does not measure recovery of causal variants in cases or review effort.

Protocols: Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, all 36 DMS sets, Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, direct assays only, Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, indirect assays only, Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions only, Livesey and Marsh 2025 DMS benchmark: pairwise |Spearman| win rate, SNV-accessible substitutions with gaps filled, Livesey and Marsh 2025 clinical benchmark: pairwise AUROC win rate, ClinVar pathogenic versus gnomAD missense

What each column measures:

• Substitutions, zero-shot: Average over proteins of AUC separating pathogenic from benign clinical substitutions
• Substitutions, supervised predictors: Average over proteins of AUC for supervised clinical predictors from dbNSFP 4.4a
• Indels, zero-shot: Average over proteins of AUC separating pathogenic from benign clinical indels

• Only 13 models are listed; the file's completeness is not stated.
• Averaging and uncertainty construction not verified for the clinical summaries.
• Clinical labels are biased towards heavily studied proteins, such as well-known cancer genes, and towards European ancestry (ProteinGym paper, Limitations).
• Evolutionary conservation is one of the criteria used to classify ClinVar variants, so even zero-shot models are not free of circularity (ProteinGym paper, Limitations).
• Per-protein AUC averaged over proteins does not test whether scores are comparable across genes.
• The ProteinGym paper describes the 2023 clinical sets; whether the sets behind the 144fe22 files are unchanged is not stated.
• The use case excludes balanced pathogenic/benign classification as a measure of case-level diagnostic performance; this protocol is variant-level classification and does not measure recovery of causal variants in cases or review effort.
• Transcribed from the source; no reproduction by Rewire and no human scientific review.
• Do not compare with the zero-shot rows as if on equal footing.
• Performance is reported on all available ClinVar labels, so supervised methods whose training data overlap these labels are likely overestimated; models trained on population frequencies leak further, because most ClinVar benign labels rest on observed frequency (ProteinGym paper, 'Clinical datasets', supervised benchmark).
• No uncertainty column in this file.
• Benign indels are common gnomAD indels used as pseudocontrols, because 84% of ClinVar indel annotations are pathogenic; the task partly separates pathogenic from common variants (ProteinGym paper, 'Clinical datasets' and Appendix A.3.2).
• Only short in-frame indels (up to 3 amino acids) are included; frameshifts were filtered out (Appendix A.3.2).

Protocols: ProteinGym clinical substitutions, zero-shot, average AUC (commit 144fe22), ProteinGym clinical substitutions, supervised predictors, average AUC (commit 144fe22), ProteinGym clinical indels, zero-shot, average AUC (commit 144fe22)

• Do not combine with Talos Table1 or infer matched equal-effort superiority.
• Minor software version and knowledge release not extracted.

Protocol: Exomiser rank-budget recovery in ACG trios