Contested result

mCSM and ThermoNet: Transfer learning to leverage larger datasets for improved prediction of protein stability changes reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Transfer learning to leverage larger datasets for improved prediction of protein stability changes ranks mCSM above ThermoNet in 6 of 6 comparisons; other studies rank ThermoNet above mCSM in 8 of 10.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Predicting protein stability changes… ranks ThermoNet above mCSM in 8 of its 10 comparisons and mCSM higher in 2. Transfer learning to leverage larger… ranks mCSM above ThermoNet in all 6 of its comparisons.

Rank ThermoNet higher (1): Predicting protein stability changes…
Rank mCSM higher (1): Transfer learning to leverage larger… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

ThermoNet higherTiemCSM higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks ThermoNet higher	Ranks mCSM higher	Ties
Transfer learning to leverage larger datasets for improved prediction of protein stability changes · DOI	Disagrees	0	6	0
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset · DOI	Agrees	8	2	0

Comparisons in Transfer learning to leverage larger… (6)

Protocol	Dataset	Metric	ThermoNet	mCSM	Higher
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	Pearson (Fireprot homologue-free split), unitless	0.37	0.59	mCSM
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	RMSE (Fireprot homologue-free split), kilocalorie-per-mole	1.99	1.53	mCSM
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	Spearman (Fireprot homologue-free split), unitless	0.36	0.57	mCSM
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	Pearson (Megascale held-out test split), unitless	0.33	0.49	mCSM
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	RMSE (Megascale held-out test split), kilocalorie-per-mole	1.2	0.97	mCSM
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	Spearman (Megascale held-out test split), unitless	0.27	0.41	mCSM

Comparisons in Predicting protein stability changes… (10)

Protocol	Dataset	Metric	ThermoNet	mCSM	Higher
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, 669 direct variants), kilocalorie-per-mole	1.17	1.13	mCSM
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, 669 reverse variants), kilocalorie-per-mole	1.23	1.86	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole	1.2	1.49	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, 669 direct variants), unitless	0.39	0.36	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, 669 reverse variants), unitless	0.38	0.22	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, all 1,338 direct and reverse variants), unitless	0.51	0.37	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better)), unitless	-0.85	-0.05	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, 669 direct variants), kilocalorie-per-mole	1.62	1.54	mCSM
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, 669 reverse variants), kilocalorie-per-mole	1.66	2.3	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole	1.64	1.96	ThermoNet

What disagrees

Other studies: rank ThermoNet above mCSM in 8 of 10 comparisons
This study: ranks mCSM above ThermoNet in 6 of 6 comparisons

Compared: mCSM, ThermoNet
Study that disagrees: Transfer learning to leverage larger datasets for improved prediction of protein stability changes
Other studies: Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (12)
Study	Tool	Protocol or benchmark	Metric	Value
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Fireprot homologue-free split)	0.59
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Fireprot homologue-free split)	1.53
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Fireprot homologue-free split)	0.57
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Megascale held-out test split)	0.49
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Megascale held-out test split)	0.97
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	mCSM (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Megascale held-out test split)	0.41
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Fireprot homologue-free split)	0.37
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Fireprot homologue-free split)	1.99
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Fireprot homologue-free split)	0.36
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Megascale held-out test split)	0.33
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Megascale held-out test split)	1.2
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Megascale held-out test split)	0.27

In the other studies (20)
Study	Tool	Protocol or benchmark	Metric	Value
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better))	–0.05
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 direct variants)	1.13
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 direct variants)	0.36
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 direct variants)	1.54
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 reverse variants)	1.86
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 reverse variants)	0.22
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 reverse variants)	2.3
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, all 1,338 direct and reverse variants)	1.49
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, all 1,338 direct and reverse variants)	0.37
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	mCSM (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, all 1,338 direct and reverse variants)	1.96
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better))	–0.85
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 direct variants)	1.17
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 direct variants)	0.39
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 direct variants)	1.62
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 reverse variants)	1.23
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 reverse variants)	0.38
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 reverse variants)	1.66
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, all 1,338 direct and reverse variants)	1.2
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, all 1,338 direct and reverse variants)	0.51
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, all 1,338 direct and reverse variants)	1.64

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-752baf99433aef14 · All contested results · How findings are made and followed up