Contested result

MUpro and ThermoNet: Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset ranks ThermoNet above MUpro in 9 of 9 comparisons; other studies rank MUpro above ThermoNet in 5 of 6.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Transfer learning to leverage larger… ranks MUpro above ThermoNet in 5 of its 6 comparisons and ThermoNet higher in 1. Predicting protein stability changes… ranks ThermoNet above MUpro in all 9 comparisons that are not ties (1 is a tie).

Rank MUpro higher (1): Transfer learning to leverage larger…
Rank ThermoNet higher (1): Predicting protein stability changes… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

MUpro higherTieThermoNet higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks MUpro higher	Ranks ThermoNet higher	Ties
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset · DOI	Disagrees	0	9	1
Transfer learning to leverage larger datasets for improved prediction of protein stability changes · DOI	Agrees	5	1	0

Comparisons in Predicting protein stability changes… (10)

Protocol	Dataset	Metric	MUpro	ThermoNet	Higher
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, 669 direct variants), kilocalorie-per-mole	1.21	1.17	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, 669 reverse variants), kilocalorie-per-mole	1.96	1.23	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole	1.58	1.2	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, 669 direct variants), unitless	0.25	0.39	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, 669 reverse variants), unitless	0.2	0.38	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, all 1,338 direct and reverse variants), unitless	0.32	0.51	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better)), unitless	-0.32	-0.85	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, 669 direct variants), kilocalorie-per-mole	1.61	1.62	Tie
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, 669 reverse variants), kilocalorie-per-mole	2.38	1.66	ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole	2.03	1.64	ThermoNet

Comparisons in Transfer learning to leverage larger… (6)

Protocol	Dataset	Metric	MUpro	ThermoNet	Higher
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	Pearson (Fireprot homologue-free split), unitless	0.58	0.37	MUpro
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	RMSE (Fireprot homologue-free split), kilocalorie-per-mole	1.54	1.99	MUpro
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	Spearman (Fireprot homologue-free split), unitless	0.57	0.36	MUpro
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	Pearson (Megascale held-out test split), unitless	0.31	0.33	ThermoNet
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	RMSE (Megascale held-out test split), kilocalorie-per-mole	1.08	1.2	MUpro
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	Spearman (Megascale held-out test split), unitless	0.29	0.27	MUpro

What disagrees

Other studies: rank MUpro above ThermoNet in 5 of 6 comparisons
This study: ranks ThermoNet above MUpro in 9 of 9 comparisons

Compared: MUpro, ThermoNet
Study that disagrees: Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset
Other studies: Transfer learning to leverage larger datasets for improved prediction of protein stability changes
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (18)
Study	Tool	Protocol or benchmark	Metric	Value
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MUpro (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better))	–0.32
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MUpro (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 direct variants)	1.21
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MUpro (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 direct variants)	0.25
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MUpro (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 reverse variants)	1.96
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MUpro (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 reverse variants)	0.2
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MUpro (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 reverse variants)	2.38
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MUpro (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, all 1,338 direct and reverse variants)	1.58
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MUpro (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, all 1,338 direct and reverse variants)	0.32
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MUpro (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, all 1,338 direct and reverse variants)	2.03
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better))	–0.85
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 direct variants)	1.17
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 direct variants)	0.39
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 reverse variants)	1.23
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 reverse variants)	0.38
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 reverse variants)	1.66
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, all 1,338 direct and reverse variants)	1.2
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, all 1,338 direct and reverse variants)	0.51
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ThermoNet (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, all 1,338 direct and reverse variants)	1.64

In the other studies (12)
Study	Tool	Protocol or benchmark	Metric	Value
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MUPRO (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Fireprot homologue-free split)	0.58
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MUPRO (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Fireprot homologue-free split)	1.54
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MUPRO (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Fireprot homologue-free split)	0.57
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MUPRO (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Megascale held-out test split)	0.31
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MUPRO (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Megascale held-out test split)	1.08
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MUPRO (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Megascale held-out test split)	0.29
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Fireprot homologue-free split)	0.37
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Fireprot homologue-free split)	1.99
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Fireprot homologue-free split)	0.36
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Megascale held-out test split)	0.33
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Megascale held-out test split)	1.2
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ThermoNet (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Megascale held-out test split)	0.27

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-e7d4daff4ea01516 · All contested results · How findings are made and followed up