Contested result

ACDC-NN and MAESTRO: Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset reverses the order

Low severity
Not yet reviewed
Ordering reversal
Different tasks across studies

Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset ranks ACDC-NN above MAESTRO in 7 of 9 comparisons; other studies rank MAESTRO above ACDC-NN in 5 of 5.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

Transfer learning to leverage larger… ranks MAESTRO above ACDC-NN in all 5 comparisons that are not ties (1 is a tie). Predicting protein stability changes… ranks ACDC-NN above MAESTRO in 7 of its 10 comparisons and MAESTRO higher in 2; 1 is a tie.

Rank MAESTRO higher (1): Transfer learning to leverage larger…
Rank ACDC-NN higher (1): Predicting protein stability changes… (disagrees)
Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

MAESTRO higherTieACDC-NN higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.

Papers compared (2)
Paper	Role	Ranks MAESTRO higher	Ranks ACDC-NN higher	Ties
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset · DOI	Disagrees	2	7	1
Transfer learning to leverage larger datasets for improved prediction of protein stability changes · DOI	Agrees	5	0	1

Comparisons in Predicting protein stability changes… (10)

Protocol	Dataset	Metric	MAESTRO	ACDC-NN	Higher
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, 669 direct variants), kilocalorie-per-mole	1.06	1.05	Tie
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, 669 reverse variants), kilocalorie-per-mole	1.66	1.06	ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	MAE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole	1.3	1.05	ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, 669 direct variants), unitless	0.5	0.46	MAESTRO
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, 669 reverse variants), unitless	0.2	0.45	ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, all 1,338 direct and reverse variants), unitless	0.44	0.61	ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better)), unitless	-0.22	-1	ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, 669 direct variants), kilocalorie-per-mole	1.44	1.49	MAESTRO
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, 669 reverse variants), kilocalorie-per-mole	2.1	1.5	ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variants	RMSE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole	1.8	1.5	ACDC-NN

Comparisons in Transfer learning to leverage larger… (6)

Protocol	Dataset	Metric	MAESTRO	ACDC-NN	Higher
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	Pearson (Fireprot homologue-free split), unitless	0.62	0.57	MAESTRO
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	RMSE (Fireprot homologue-free split), kilocalorie-per-mole	1.49	1.69	MAESTRO
Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	FireProtDB homologue-free split with experimental structures	Spearman (Fireprot homologue-free split), unitless	0.6	0.51	MAESTRO
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	Pearson (Megascale held-out test split), unitless	0.55	0.52	MAESTRO
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	RMSE (Megascale held-out test split), kilocalorie-per-mole	1.04	1.05	Tie
Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Megascale (Tsuboyama 2023) test split, homology-clustered at 25% identity	Spearman (Megascale held-out test split), unitless	0.47	0.45	MAESTRO

What disagrees

Other studies: rank MAESTRO above ACDC-NN in 5 of 5 comparisons
This study: ranks ACDC-NN above MAESTRO in 7 of 9 comparisons

Compared: ACDC-NN, MAESTRO
Study that disagrees: Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset
Other studies: Transfer learning to leverage larger datasets for improved prediction of protein stability changes
Rule: Ordering reversal; different tasks across studies
Model identity: Matched across studies partly through links not yet reviewed

The compared numbers

Each value links to its result record and source. Compare values within one protocol; the rule compares rankings, not raw values across protocols.

In the study that disagrees (18)
Study	Tool	Protocol or benchmark	Metric	Value
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ACDC-NN (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 direct variants)	0.46
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ACDC-NN (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 direct variants)	1.49
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ACDC-NN (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 reverse variants)	1.06
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ACDC-NN (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 reverse variants)	0.45
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ACDC-NN (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 reverse variants)	1.5
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ACDC-NN (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, all 1,338 direct and reverse variants)	1.05
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ACDC-NN (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, all 1,338 direct and reverse variants)	0.61
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ACDC-NN (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, all 1,338 direct and reverse variants)	1.5
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	ACDC-NN-Seq (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better))	–1
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MAESTRO (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better))	–0.22
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MAESTRO (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 direct variants)	0.5
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MAESTRO (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 direct variants)	1.44
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MAESTRO (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, 669 reverse variants)	1.66
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MAESTRO (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, 669 reverse variants)	0.2
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MAESTRO (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, 669 reverse variants)	2.1
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MAESTRO (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	MAE (S669, all 1,338 direct and reverse variants)	1.3
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MAESTRO (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	Pearson (S669, all 1,338 direct and reverse variants)	0.44
Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset	MAESTRO (Pancotti et al. 2022)	S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)	RMSE (S669, all 1,338 direct and reverse variants)	1.8

In the other studies (10)
Study	Tool	Protocol or benchmark	Metric	Value
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ACDC-NN (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Fireprot homologue-free split)	0.57
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ACDC-NN (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Fireprot homologue-free split)	1.69
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ACDC-NN (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Fireprot homologue-free split)	0.51
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ACDC-NN (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Megascale held-out test split)	0.52
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	ACDC-NN (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Megascale held-out test split)	0.45
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MAESTRO (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Fireprot homologue-free split)	0.62
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MAESTRO (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	RMSE (Fireprot homologue-free split)	1.49
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MAESTRO (Dieckhaus et al. 2024)	Fireprot homologue-free split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Fireprot homologue-free split)	0.6
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MAESTRO (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Pearson (Megascale held-out test split)	0.55
Transfer learning to leverage larger datasets for improved prediction of protein stability changes	MAESTRO (Dieckhaus et al. 2024)	Megascale held-out test split ddG prediction (Dieckhaus et al. 2024 Table 2)	Spearman (Megascale held-out test split)	0.47

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-b320f1a448d56b17 · All contested results · How findings are made and followed up