rewirebio.iobenchmarks
Contested result

MUpro and ThermoNet: Transfer learning to leverage larger datasets for improved prediction of protein stability changes and Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Transfer learning to leverage larger datasets for improved prediction of protein stability changes ranks MUpro above ThermoNet in 5 of 6 comparisons; Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset ranks ThermoNet above MUpro in 9 of 9 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank MUpro above ThermoNet in 5 of their 16 comparisons and ThermoNet higher in 10; 1 is a tie. Of those papers, Predicting protein stability changes… ranks ThermoNet higher on its own (9 of 10).

Papers grouped by which system they rank higher: MUpro or ThermoNetRank MUpro higher: Transfer learning to leverage larger…. Rank ThermoNet higher: Predicting protein stability changes…. Mixed: none.Rank MUpro higherMixedRank ThermoNet higherTransfer learningto leverage larger…Transfer learning to leverage larger datasets for improved prediction of protein stability changesPredicting proteinstability changes…Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset
  • Rank MUpro higher (1): Transfer learning to leverage larger…
  • Rank ThermoNet higher (1): Predicting protein stability changes…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores MUpro or ThermoNet higherPredicting protein stability changes…: MUpro higher in 0, ThermoNet higher in 9, 1 tie. Transfer learning to leverage larger…: MUpro higher in 5, ThermoNet higher in 1, 0 ties.Predicting protein stability changes…Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset1 tie19 ThermoNet higher9Transfer learning to leverage larger…Transfer learning to leverage larger datasets for improved prediction of protein stability changes5 MUpro higher51 ThermoNet higher1
MUpro higherTieThermoNet higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Predicting protein stability changes… (10)
ProtocolDatasetMetricMUproThermoNetHigher
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsMAE (S669, 669 direct variants), kilocalorie-per-mole1.211.17ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsMAE (S669, 669 reverse variants), kilocalorie-per-mole1.961.23ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsMAE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole1.581.2ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, 669 direct variants), unitless0.250.39ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, 669 reverse variants), unitless0.20.38ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, all 1,338 direct and reverse variants), unitless0.320.51ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better)), unitless-0.32-0.85ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsRMSE (S669, 669 direct variants), kilocalorie-per-mole1.611.62Tie
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsRMSE (S669, 669 reverse variants), kilocalorie-per-mole2.381.66ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsRMSE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole2.031.64ThermoNet
Comparisons in Transfer learning to leverage larger… (6)

What disagrees

studies
2
Compared
MUpro, ThermoNet
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-a9186e5d3eddd879 · All contested results · How findings are made and followed up