rewirebio.iobenchmarks
Contested result

ACDC-NN and MAESTRO: Transfer learning to leverage larger datasets for improved prediction of protein stability changes and Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • unknown

Transfer learning to leverage larger datasets for improved prediction of protein stability changes ranks MAESTRO above ACDC-NN in 5 of 5 comparisons; Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset ranks ACDC-NN above MAESTRO in 7 of 9 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank ACDC-NN above MAESTRO in 7 of their 16 comparisons and MAESTRO higher in 7; 2 are ties. Of those papers, Transfer learning to leverage larger… ranks MAESTRO higher on its own (5 of 6).

Papers grouped by which system they rank higher: ACDC-NN or MAESTRORank ACDC-NN higher: Predicting protein stability changes…. Rank MAESTRO higher: Transfer learning to leverage larger…. Mixed: none.Rank ACDC-NN higherMixedRank MAESTRO higherPredicting proteinstability changes…Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new datasetTransfer learningto leverage larger…Transfer learning to leverage larger datasets for improved prediction of protein stability changes
  • Rank ACDC-NN higher (1): Predicting protein stability changes…
  • Rank MAESTRO higher (1): Transfer learning to leverage larger…
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores ACDC-NN or MAESTRO higherPredicting protein stability changes…: ACDC-NN higher in 7, MAESTRO higher in 2, 1 tie. Transfer learning to leverage larger…: ACDC-NN higher in 0, MAESTRO higher in 5, 1 tie.Predicting protein stability changes…Predicting protein stability changes upon single-point mutation: a thorough comparison of the available tools on a new dataset7 ACDC-NN higher71 tie12 MAESTRO higher2Transfer learning to leverage larger…Transfer learning to leverage larger datasets for improved prediction of protein stability changes1 tie15 MAESTRO higher5
ACDC-NN higherTieMAESTRO higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Comparisons in Predicting protein stability changes… (10)
ProtocolDatasetMetricACDC-NNMAESTROHigher
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsMAE (S669, 669 direct variants), kilocalorie-per-mole1.051.06Tie
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsMAE (S669, 669 reverse variants), kilocalorie-per-mole1.061.66ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsMAE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole1.051.3ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, 669 direct variants), unitless0.460.5MAESTRO
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, 669 reverse variants), unitless0.450.2ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, all 1,338 direct and reverse variants), unitless0.610.44ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsPearson (S669, antisymmetry, direct versus reverse predictions (perfect antisymmetry is -1, so lower is better)), unitless-0.98-0.22ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsRMSE (S669, 669 direct variants), kilocalorie-per-mole1.491.44MAESTRO
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsRMSE (S669, 669 reverse variants), kilocalorie-per-mole1.52.1ACDC-NN
S669 direct and reverse ddG prediction (Pancotti et al. 2022 Table 1)S669: 669 single-point variants from ThermoMutDB in proteins under 25% identity to S2648 and VariBench, with reverse variantsRMSE (S669, all 1,338 direct and reverse variants), kilocalorie-per-mole1.51.8ACDC-NN
Comparisons in Transfer learning to leverage larger… (6)

What disagrees

studies
2
Compared
ACDC-NN, MAESTRO
Study that disagrees
Other studies
Rule
Two-study conflict; unknown
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-c2045a4947684849 · All contested results · How findings are made and followed up