Contested result

mCSM and ThermoNet: Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset ranks ThermoNet above mCSM in 8 of 10 comparisons; other studies rank mCSM above ThermoNet in 6 of 6.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Transfer learning to leverage larger… ranks mCSM above ThermoNet in all 6 of its comparisons. Predicting protein stability changes… ranks ThermoNet above mCSM in 8 of its 10 comparisons and mCSM higher in 2.

Rank mCSM higher (1): Transfer learning to leverage larger…
Rank ThermoNet higher (1): Predicting protein stability changes… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

mCSM higherTieThermoNet higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks mCSM higher	Ranks ThermoNet higher	Ties
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset · DOI	Disagrees	2	8	0
Transfer learning to leverage larger datasets for improved prediction of protein stability changes · DOI	Agrees	6	0	0

Comparisons in Predicting protein stability changes… (10)

Protocol	Dataset	Metric	mCSM	ThermoNet	Higher
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, 669 direct variants), kilocalorie-per-mole	1.13	1.17	mCSM
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, 669 reverse variants), kilocalorie-per-mole	1.86	1.23	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole	1.49	1.2	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, 669 direct variants), unitless	0.36	0.39	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, 669 reverse variants), unitless	0.22	0.38	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, all 1,338 direct and reverse variants), unitless	0.37	0.51	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better)), unitless	-0.05	-0.85	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, 669 direct variants), kilocalorie-per-mole	1.54	1.62	mCSM
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, 669 reverse variants), kilocalorie-per-mole	2.3	1.66	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole	1.96	1.64	ThermoNet

Comparisons in Transfer learning to leverage larger… (6)

Protocol	Dataset	Metric	mCSM	ThermoNet	Higher
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	Pearson (Fireprot homologue-free split), unitless	0.59	0.37	mCSM
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	RMSE (Fireprot homologue-free split), kilocalorie-per-mole	1.53	1.99	mCSM
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	Spearman (Fireprot homologue-free split), unitless	0.57	0.36	mCSM
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	Pearson (Megascale held-out test split), unitless	0.49	0.33	mCSM
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	RMSE (Megascale held-out test split), kilocalorie-per-mole	0.97	1.2	mCSM
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	Spearman (Megascale held-out test split), unitless	0.41	0.27	mCSM

What disagrees

Other studies: rank mCSM above ThermoNet in 6 of 6 comparisons
This study: ranks ThermoNet above mCSM in 8 of 10 comparisons

Compared: mCSM, ThermoNet
Study that disagrees: Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset
Other studies: Transfer learning to leverage larger datasets for improved prediction of protein stability changes
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (20)
Study	Tool	Protocol or benchmark	Metric	Value
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better))	–0.05
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 direct variants)	1.13
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 direct variants)	0.36
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 direct variants)	1.54
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 reverse variants)	1.86
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 reverse variants)	0.22
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 reverse variants)	2.3
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, all 1,338 direct and reverse variants)	1.49
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, all 1,338 direct and reverse variants)	0.37
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, all 1,338 direct and reverse variants)	1.96
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better))	–0.85
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 direct variants)	1.17
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 direct variants)	0.39
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 direct variants)	1.62
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 reverse variants)	1.23
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 reverse variants)	0.38
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 reverse variants)	1.66
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, all 1,338 direct and reverse variants)	1.2
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, all 1,338 direct and reverse variants)	0.51
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, all 1,338 direct and reverse variants)	1.64

In the other studies (12)
Study	Tool	Protocol or benchmark	Metric	Value
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Fireprot homologue-free split)	0.59
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Fireprot homologue-free split)	1.53
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Fireprot homologue-free split)	0.57
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Megascale held-out test split)	0.49
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Megascale held-out test split)	0.97
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Megascale held-out test split)	0.41
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Fireprot homologue-free split)	0.37
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Fireprot homologue-free split)	1.99
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Fireprot homologue-free split)	0.36
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Megascale held-out test split)	0.33
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Megascale held-out test split)	1.2
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Megascale held-out test split)	0.27

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-e7e5851082f2e1fc · All contested results · How findings are made and followed up