rewirebio.iobenchmarks
Contested result

Boltz-2 and Protenix: Shen et al. 2025, Chemical Science (Europe PMC full text) and POISK Explorer project site disagree

  • Low severity
  • Not yet reviewed
  • Two-study conflict
  • different-task

Shen et al. 2025, Chemical Science (Europe PMC full text) ranks Boltz-2 above Protenix in 26 of 28 comparisons; POISK Explorer project site ranks Protenix above Boltz-2 in 10 of 10 comparisons. No other study gives a verdict, so neither is the outlier.

An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.

Which studies agree

2 papers rank Boltz-2 above Protenix in 26 of their 42 comparisons and Protenix higher in 12; 4 are ties. Of those papers, POISK Explorer project site ranks Protenix higher on its own (10 of 10).

Papers grouped by which system they rank higher: Boltz-2 or ProtenixRank Boltz-2 higher: Shen et al. 2025. Rank Protenix higher: POISK Explorer project site. Mixed: none.Rank Boltz-2 higherMixedRank Protenix higherShen et al. 2025Shen et al. 2025, Chemical Science (Europe PMC full text)POISK Explorer project sitePOISK Explorer project site
  • Rank Boltz-2 higher (1): Shen et al. 2025
  • Rank Protenix higher (1): POISK Explorer project site
  • Mixed (0): none

Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.

Comparisons in which each paper scores Boltz-2 or Protenix higherShen et al. 2025: Boltz-2 higher in 26, Protenix higher in 2, 4 ties. POISK Explorer project site: Boltz-2 higher in 0, Protenix higher in 10, 0 ties.Shen et al. 2025Shen et al. 2025, Chemical Science (Europe PMC full text)26 Boltz-2 higher264 tie42 Protenix higherPOISK Explorer project sitePOISK Explorer project site10 Protenix higher10
Boltz-2 higherTieProtenix higherNumbers are counts of comparisons; the outlined bar is the paper that disagrees.
Papers compared (2)
PaperRoleRanks Boltz-2 higherRanks Protenix higherTies
Shen et al. 2025, Chemical Science (Europe PMC full text) · DOIAgrees2624
POISK Explorer project site and 4 more records · DOIAgrees0100
Comparisons in Shen et al. 2025 (32)
ProtocolDatasetMetricBoltz-2ProtenixHigher
DEKOIS 2.0 screening with co-folding confidence scores against docking tools (Shen et al. 2025 Table 1)DEKOIS 2.0, 79 targets as scored by Shen et al. 2025AUROC (AUROC, mean over the 79 DEKOIS 2.0 targets), unitless0.860.795Boltz-2
DEKOIS 2.0 screening with co-folding confidence scores against docking tools (Shen et al. 2025 Table 1)DEKOIS 2.0, 79 targets as scored by Shen et al. 2025AUROC (AUROC, median over the 79 DEKOIS 2.0 targets), unitless0.8940.837Boltz-2
DEKOIS 2.0 screening with co-folding confidence scores against docking tools (Shen et al. 2025 Table 1)DEKOIS 2.0, 79 targets as scored by Shen et al. 2025Bedroc (BEDROC with alpha = 80.5, mean over the 79 DEKOIS 2.0 targets), unitless0.560.507Boltz-2
DEKOIS 2.0 screening with co-folding confidence scores against docking tools (Shen et al. 2025 Table 1)DEKOIS 2.0, 79 targets as scored by Shen et al. 2025Bedroc (BEDROC with alpha = 80.5, median over the 79 DEKOIS 2.0 targets), unitless0.6180.553Boltz-2
DEKOIS 2.0 screening with co-folding confidence scores against docking tools (Shen et al. 2025 Table 1)DEKOIS 2.0, 79 targets as scored by Shen et al. 2025Enrichment factor (EF at the top 0.5% of the ranked library, mean over the 79 DEKOIS 2.0 targets), unitless19.218.4Boltz-2
DEKOIS 2.0 screening with co-folding confidence scores against docking tools (Shen et al. 2025 Table 1)DEKOIS 2.0, 79 targets as scored by Shen et al. 2025Enrichment factor (EF at the top 0.5% of the ranked library, median over the 79 DEKOIS 2.0 targets), unitless22.122.1Tie
DEKOIS 2.0 screening with co-folding confidence scores against docking tools (Shen et al. 2025 Table 1)DEKOIS 2.0, 79 targets as scored by Shen et al. 2025Enrichment factor (EF at the top 1% of the ranked library, mean over the 79 DEKOIS 2.0 targets), unitless1816.8Boltz-2
DEKOIS 2.0 screening with co-folding confidence scores against docking tools (Shen et al. 2025 Table 1)DEKOIS 2.0, 79 targets as scored by Shen et al. 2025Enrichment factor (EF at the top 1% of the ranked library, median over the 79 DEKOIS 2.0 targets), unitless19.116.7Boltz-2
DEKOIS 2.0 screening with co-folding confidence scores against docking tools (Shen et al. 2025 Table 1)DEKOIS 2.0, 79 targets as scored by Shen et al. 2025Enrichment factor (EF at the top 5% of the ranked library, mean over the 79 DEKOIS 2.0 targets), unitless10.38.84Boltz-2
DEKOIS 2.0 screening with co-folding confidence scores against docking tools (Shen et al. 2025 Table 1)DEKOIS 2.0, 79 targets as scored by Shen et al. 2025Enrichment factor (EF at the top 5% of the ranked library, median over the 79 DEKOIS 2.0 targets), unitless119.5Boltz-2
GPCRrecent screening (16 GPCR targets) (Shen et al. 2025 Table 3)GPCRrecent, 16 GPCR targetsAUROC (AUROC, mean over the 16 GPCRrecent targets), unitless0.7380.623Boltz-2
GPCRrecent screening (16 GPCR targets) (Shen et al. 2025 Table 3)GPCRrecent, 16 GPCR targetsAUROC (AUROC, median over the 16 GPCRrecent targets), unitless0.730.632Boltz-2
GPCRrecent screening (16 GPCR targets) (Shen et al. 2025 Table 3)GPCRrecent, 16 GPCR targetsBedroc (BEDROC with alpha = 80.5, mean over the 16 GPCRrecent targets), unitless0.2360.135Boltz-2
GPCRrecent screening (16 GPCR targets) (Shen et al. 2025 Table 3)GPCRrecent, 16 GPCR targetsBedroc (BEDROC with alpha = 80.5, median over the 16 GPCRrecent targets), unitless0.1980.105Boltz-2
GPCRrecent screening (16 GPCR targets) (Shen et al. 2025 Table 3)GPCRrecent, 16 GPCR targetsEnrichment factor (EF at the top 0.5% of the ranked library, mean over the 16 GPCRrecent targets), unitless14.67.99Boltz-2
GPCRrecent screening (16 GPCR targets) (Shen et al. 2025 Table 3)GPCRrecent, 16 GPCR targetsEnrichment factor (EF at the top 0.5% of the ranked library, median over the 16 GPCRrecent targets), unitless12.95.3Boltz-2
GPCRrecent screening (16 GPCR targets) (Shen et al. 2025 Table 3)GPCRrecent, 16 GPCR targetsEnrichment factor (EF at the top 1% of the ranked library, mean over the 16 GPCRrecent targets), unitless126.97Boltz-2
GPCRrecent screening (16 GPCR targets) (Shen et al. 2025 Table 3)GPCRrecent, 16 GPCR targetsEnrichment factor (EF at the top 1% of the ranked library, median over the 16 GPCRrecent targets), unitless9.35.67Boltz-2
GPCRrecent screening (16 GPCR targets) (Shen et al. 2025 Table 3)GPCRrecent, 16 GPCR targetsEnrichment factor (EF at the top 5% of the ranked library, mean over the 16 GPCRrecent targets), unitless5.73.59Boltz-2
GPCRrecent screening (16 GPCR targets) (Shen et al. 2025 Table 3)GPCRrecent, 16 GPCR targetsEnrichment factor (EF at the top 5% of the ranked library, median over the 16 GPCRrecent targets), unitless5.032.88Boltz-2
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Enrichment factor (EF at the top 1% of the ranked library, LIT-PCBA target ESR_ago), unitless7.697.69Tie
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Enrichment factor (EF at the top 1% of the ranked library, LIT-PCBA target ESR_antago), unitless3.881.94Boltz-2
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Enrichment factor (EF at the top 1% of the ranked library, LIT-PCBA target MAPK1), unitless3.241.3Boltz-2
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Enrichment factor (EF at the top 1% of the ranked library, LIT-PCBA target PPARG), unitless25.622Boltz-2
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Enrichment factor (EF at the top 1% of the ranked library, LIT-PCBA target TP53), unitless05Protenix
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Enrichment factor (EF at the top 1% of the ranked library, averaged over the five targets), unitless8.097.58Boltz-2
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Normalized enrichment factor (NEF at the top 1% of the ranked library, LIT-PCBA target ESR_ago), unitless0.0770.077Tie
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Normalized enrichment factor (NEF at the top 1% of the ranked library, LIT-PCBA target ESR_antago), unitless0.0780.039Boltz-2
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Normalized enrichment factor (NEF at the top 1% of the ranked library, LIT-PCBA target MAPK1), unitless0.0320.013Boltz-2
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Normalized enrichment factor (NEF at the top 1% of the ranked library, LIT-PCBA target PPARG), unitless0.2590.222Boltz-2
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Normalized enrichment factor (NEF at the top 1% of the ranked library, LIT-PCBA target TP53), unitless00.093Protenix
LIT-PCBA five-target screening (Shen et al. 2025 Table 4)LIT-PCBA five-target subset as scored by Shen et al. 2025Normalized enrichment factor (NEF at the top 1% of the ranked library, averaged over the five targets), unitless0.0890.089Tie
Comparisons in POISK Explorer project site (10)
ProtocolDatasetMetricBoltz-2ProtenixHigher
POISK binder-decoy separation in full-Fv ipSAE, Cohen's dPOISK co-folding binder-decoy benchmark set (22,494 complexes)Cohen's d (separation of binder and decoy full-Fv ipSAE score distributions over all 22,494 complexes), unitless0.230.88Protenix
POISK binder-decoy separation in ipTM, Cohen's dPOISK co-folding binder-decoy benchmark set (22,494 complexes)Cohen's d (separation of binder and decoy ipTM score distributions over all 22,494 complexes), unitless0.20.75Protenix
POISK epitope contact recoveryPOISK binder records with mappable patent epitope annotations (1,232)Success rate (at least one annotated epitope residue with a heavy atom within 4 angstrom of an antibody heavy atom, over 1,232 binder records with mappable patent epitope annotations), percent55.462.8Protenix
POISK pairwise binder-versus-decoy accuracy at 10% coveragePOISK co-folding binder-decoy benchmark set (22,494 complexes)Paired accuracy (binder scored above its matched putative decoy by full-Fv ipSAE, top 10% of the 11,247 pairs ranked by within-pair maximum ipSAE, exact ties at half credit), fraction0.6410.978Protenix
POISK pairwise binder-versus-decoy accuracy at 100% coveragePOISK co-folding binder-decoy benchmark set (22,494 complexes)Paired accuracy (binder scored above its matched putative decoy by full-Fv ipSAE, 100% coverage of the 11,247 pairs, exact ties at half credit), fraction0.5810.719Protenix
POISK pairwise binder-versus-decoy accuracy at 25% coveragePOISK co-folding binder-decoy benchmark set (22,494 complexes)Paired accuracy (binder scored above its matched putative decoy by full-Fv ipSAE, top 25% of the 11,247 pairs ranked by within-pair maximum ipSAE, exact ties at half credit), fraction0.6220.948Protenix
POISK pairwise binder-versus-decoy accuracy at 50% coveragePOISK co-folding binder-decoy benchmark set (22,494 complexes)Paired accuracy (binder scored above its matched putative decoy by full-Fv ipSAE, top 50% of the 11,247 pairs ranked by within-pair maximum ipSAE, exact ties at half credit), fraction0.6020.87Protenix
POISK pairwise binder-versus-decoy accuracy at 75% coveragePOISK co-folding binder-decoy benchmark set (22,494 complexes)Paired accuracy (binder scored above its matched putative decoy by full-Fv ipSAE, top 75% of the 11,247 pairs ranked by within-pair maximum ipSAE, exact ties at half credit), fraction0.5910.781Protenix
POISK 1-in-50 screening simulation, top-1 retrievalPOISK 1-in-50 screening targets (27 antigen constructs)Top-1 accuracy (retrieval; one patent-supported binder ranked against 49 putative decoys for the same antigen by full-Fv ipSAE, mean over 27 antigen constructs, 20 trials per construct per seed, 25 seeds), fraction0.030.29Protenix
POISK 1-in-50 screening simulation, top-1 retrievalPOISK 1-in-50 screening targets (27 antigen constructs)Top-1 accuracy (retrieval; one patent-supported binder ranked against 49 putative decoys for the same antigen by full-Fv ipSAE, mean over 27 antigen constructs, 20 trials per construct per seed, 25 seeds), percent3.128.9Protenix

What disagrees

studies
2
Compared
Boltz-2, Protenix
Study that disagrees
Other studies
Rule
Two-study conflict; different-task
Model identity
Matched across studies partly through links not yet reviewed

Follow-up

Current state: Not yet reviewed.

No one has reviewed this finding yet.

How to help resolve this

If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.

Finding discrepancy-1593d9dbfee36f07 · All contested results · How findings are made and followed up