rewirebio.iobenchmarks
Contested result

mCSM and ThermoNet: Transfer learning to leverage larger datasets for improved prediction of protein stability changes and Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Transfer learning to leverage larger datasets for improved prediction of protein stability changes ranks mCSM above ThermoNet in 6 of 6 comparisons; Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset ranks ThermoNet above mCSM in 8 of 10 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank mCSM above ThermoNet in 8 of their 16 comparisons and ThermoNet higher in 8. Of those papers, Predicting protein stability changes… ranks ThermoNet higher on its own (8 of 10).

Papers grouped by which system they rank higher: mCSM or ThermoNetRank mCSM higher: Transfer learning to leverage larger…. Rank ThermoNet higher: Predicting protein stability changes…. Mixed: none.Rank mCSM higherMixedRank ThermoNet higherTransfer learningto leverage larger…Transfer learning to leverage larger datasets for improved prediction of protein stability changesPredicting proteinstability changes…Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset
  • Rank mCSM higher (1): Transfer learning to leverage larger…
  • Rank ThermoNet higher (1): Predicting protein stability changes…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores mCSM or ThermoNet higherPredicting protein stability changes…: mCSM higher in 2, ThermoNet higher in 8, 0 ties. Transfer learning to leverage larger…: mCSM higher in 6, ThermoNet higher in 0, 0 ties.Predicting protein stability changes…Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset2 mCSM higher28 ThermoNet higher8Transfer learning to leverage larger…Transfer learning to leverage larger datasets for improved prediction of protein stability changes6 mCSM higher6
mCSM higherTieThermoNet higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Predicting protein stability changes… (10)
ProtocolDatasetMetricmCSMThermoNetHigher
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsMAE (S669, 669 direct variants), kilocalorie-per-mole1.131.17mCSM
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsMAE (S669, 669 reverse variants), kilocalorie-per-mole1.861.23ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsMAE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole1.491.2ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, 669 direct variants), unitless0.360.39ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, 669 reverse variants), unitless0.220.38ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, all 1,338 direct and reverse variants), unitless0.370.51ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better)), unitless-0.05-0.85ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsRMSE (S669, 669 direct variants), kilocalorie-per-mole1.541.62mCSM
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsRMSE (S669, 669 reverse variants), kilocalorie-per-mole2.31.66ThermoNet
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsRMSE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole1.961.64ThermoNet
Comparisons in Transfer learning to leverage larger… (6)

What disagrees

studies
2
Compared
mCSM, ThermoNet
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-d63d8f9766fef049 · All contested results · How findings are made and followed up