MSA_Transformer (Livesey and Marsh 2025); Predictions from the ProteinGym website (Table S2 'Source'); ProteinGym release not stated

Overview

An explanatory profile has not yet been reviewed for this record.

Consult the linked sources for architecture or protocol details. Missing evidence is not evidence of a missing capability.

Evaluations and results

5 evaluations · 6 results. Different protocols are not a single leaderboard.

Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)

Sorted by Pairwise win rate (average pairwise win rate against all other VEPs, |Spearman| with DMS on shared variants, 36 human DMS sets) (higher is better). The best value in each column is highlighted. Decimals are rounded for display; each value links to the printed value and its source.

Tested configuration	Protocol and dataset	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only)	Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score)	Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets)
MSA_Transformer (Livesey and Marsh 2025)Evaluation	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets	75.9%	Not reported	Not reported	Not reported	Not reported	24.4%
MSA_Transformer (Livesey and Marsh 2025)Evaluation	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only	Not reported	77.7%	Not reported	Not reported	Not reported	Not reported
MSA_Transformer (Livesey and Marsh 2025)Evaluation	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only	Not reported	Not reported	74.7%	Not reported	Not reported	Not reported
MSA_Transformer (Livesey and Marsh 2025)Evaluation	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled	Not reported	Not reported	Not reported	Not reported	72%	Not reported
MSA_Transformer (Livesey and Marsh 2025)Evaluation	Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only	Not reported	Not reported	Not reported	79.2%	Not reported	Not reported

All 6 result rows with coverage, uncertainty and sources

Filter evaluations

Tested model or methodProtocolDatasetEvaluation setupMetricEvidence origin

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configuration	Protocol and dataset	Finding	Evidence and details
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	75.9% Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: 28/36 human DMS datasets (one per protein) with predictions for this VEP scored ('proteins' column, cell CV18)	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S3', cell CU18 (row 'MSA_Transformer', column 'rank_score')
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	24.4% Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: Not reported scored / Not reported eligible	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S3', cell CW18 (row 'MSA_Transformer', column 'top_5_winrate')
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	77.7% Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: Not reported scored / Not reported eligible	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S5', cell C18 (row 'MSA_Transformer', column 'rank_score_direct')
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	74.7% Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: Not reported scored / Not reported eligible	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S5', cell B18 (row 'MSA_Transformer', column 'rank_score_indirect')
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	72% Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: 28/36 human DMS datasets (one per protein) with predictions for this VEP scored ('proteins' column, cell C18)	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S7', cell B18 (row 'MSA_Transformer', column 'rank_score')
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	79.2% Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: 28/36 human DMS datasets (one per protein) with predictions for this VEP scored ('proteins' column, cell C13)	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S6', cell B13 (row 'MSA_Transformer', column 'rank_score')

Source checking is not independent reproduction. Release 2026-10-22-1b7cf027e4f6.

Use this configuration

How it works, versions and access

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

0 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-10-22-1b7cf027e4f6
Property and statement	Original source and location	Review and provenance

No evidence rows match these filters. Choose another scope or clear the search.

How to read the evidence

Sources and history

Release 2026-10-22-1b7cf027e4f6 · Record review: source checked

2 source records and release history

Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8) · Original source · 13059_2025_3575_MOESM2_ESM.xlsx as published with PMC12016141
Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Original source · PMC12016141 (published 2025-04-22; Genome Biol 26:104)

Download this release (gzip)

Technical metadata and extraction receipts

Stable ID: vep-lit-20261014-config-livesey2025-msa-transformer

areas: dna-genomes; proteins-complexes
contexts: research; clinical_research
method types: foundation_model
reported name: MSA_Transformer
foundation model eligible: true
configuration type: paper_evaluation
version: Predictions from the ProteinGym website (Table S2 'Source'); ProteinGym release not stated
training overlap: Livesey and Marsh 2025 type: Population-free
access reuse: Livesey and Marsh obtained predictions from: ProteinGym
source locator: Supplementary Material 2, Table S2 row 49 ('MSA Transformer'); printed label 'MSA_Transformer' in Tables S3 and S5-S8
missing metadata: checkpoint: reason: unreported; note: Model size or checkpoint not stated in the paper
review: reviewer: claude; reviewer note: Claude review agent; no human review claimed. Independent Claude (Opus 5.5) review agent, separate from the extraction agent that built this batch. Automated review only. Transcription, not reproduction.; reviewed at: 2026-10-14T15:30:00Z; artifact sha256: 1b0d792d1a5d3daa95545a7973cabb131c45e42c952e7e5a447a72deef1d1e70; retrieval url: https://static-content.springer.com/esm/art%3A10.1186%2Fs13059-025-03575-w/MediaObjects/13059_2025_3575_MOESM2_ESM.xlsx; method: ai-assisted-source-review; scope-review; method note: Descriptive fields, identity links and missing-metadata reasons checked against the cited sources.; note: Checked; see the dated review for rulings on identity and method type.
provenance note: MSA_Transformer as scored in the Livesey and Marsh 2025 VEP benchmark.

Related records

Tested configuration	Protocol and dataset	Finding	Evidence and details
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	75.9% Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, 36 human DMS sets) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: 28/36 human DMS datasets (one per protein) with predictions for this VEP scored ('proteins' column, cell CV18)	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S3', cell CU18 (row 'MSA_Transformer', column 'rank_score')
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	24.4% Pairwise win rate (average pairwise win rate against the five top-ranked VEPs (column top_5_winrate; the five are not listed in the table), \|Spearman\| with DMS on shared variants, 36 human DMS sets) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: Not reported scored / Not reported eligible	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, all 36 DMS sets Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S3', cell CW18 (row 'MSA_Transformer', column 'top_5_winrate')
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	77.7% Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, direct DMS assays only) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: Not reported scored / Not reported eligible	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, direct assays only Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S5', cell C18 (row 'MSA_Transformer', column 'rank_score_direct')
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	74.7% Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, indirect DMS assays only) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: Not reported scored / Not reported eligible	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, indirect assays only Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S5', cell B18 (row 'MSA_Transformer', column 'rank_score_indirect')
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	72% Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS, SNV-accessible substitutions, missing predictions filled with the VEP's most benign score) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: 28/36 human DMS datasets (one per protein) with predictions for this VEP scored ('proteins' column, cell C18)	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions with gaps filled Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S7', cell B18 (row 'MSA_Transformer', column 'rank_score')
Configuration: MSA_Transformer (Livesey and Marsh 2025)	Protocol: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only Dataset: Livesey and Marsh 2025 human DMS benchmark set (36 proteins)	79.2% Pairwise win rate (average pairwise win rate against all other VEPs, \|Spearman\| with DMS on shared variants, SNV-accessible substitutions only) percent · higher Uncertainty: Not reported by the source: Bootstrap standard deviations appear only as Figure 2 error bars; not printed Coverage: 28/36 human DMS datasets (one per protein) with predictions for this VEP scored ('proteins' column, cell C13)	Independent external evaluation · Source checked Methods, coverage and source MSA_Transformer: Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only Livesey and Marsh 2025 DMS benchmark: pairwise \|Spearman\| win rate, SNV-accessible substitutions only Aggregation: Not reported Livesey and Marsh 2025, Supplementary Material 2 (Tables S1-S8); Livesey and Marsh 2025, Genome Biology 26:104 (Europe PMC full text XML) · Supplementary Material 2, sheet 'Table S6', cell B13 (row 'MSA_Transformer', column 'rank_score')