Boltz-2 and Protenix: Shen et al. 2025, Chemical Science (Europe PMC full text) and POISK Explorer project site disagree
- Low severity
- Not yet reviewed
- Two-study conflict
- different-task
Shen et al. 2025, Chemical Science (Europe PMC full text) ranks Boltz-2 above Protenix in 26 of 28 comparisons; POISK Explorer project site ranks Protenix above Boltz-2 in 10 of 10 comparisons. No other study gives a verdict, so neither is the outlier.
An automatic lead, not a finding of error. The results below are shown as published, everywhere on the site, until a reviewer decides otherwise.
Which studies agree
2 papers rank Boltz-2 above Protenix in 26 of their 42 comparisons and Protenix higher in 12; 4 are ties. Of those papers, POISK Explorer project site ranks Protenix higher on its own (10 of 10).
- Rank Boltz-2 higher (1): Shen et al. 2025
- Rank Protenix higher (1): POISK Explorer project site
- Mixed (0): none
Each paper sits where at least two thirds of its comparisons point; Mixed means neither model reaches two thirds. The larger, outlined dot is the paper that disagrees.
| Paper | Role | Ranks Boltz-2 higher | Ranks Protenix higher | Ties |
|---|---|---|---|---|
| Shen et al. 2025, Chemical Science (Europe PMC full text) · DOI | Agrees | 26 | 2 | 4 |
| POISK Explorer project site and 4 more records · DOI | Agrees | 0 | 10 | 0 |
Comparisons in Shen et al. 2025 (32)
Comparisons in POISK Explorer project site (10)
| Protocol | Dataset | Metric | Boltz-2 | Protenix | Higher |
|---|---|---|---|---|---|
| POISK binder-decoy separation in full-Fv ipSAE, Cohen's d | POISK co-folding binder-decoy benchmark set (22,494 complexes) | Cohen's d (separation of binder and decoy full-Fv ipSAE score distributions over all 22,494 complexes), unitless | 0.23 | 0.88 | Protenix |
| POISK binder-decoy separation in ipTM, Cohen's d | POISK co-folding binder-decoy benchmark set (22,494 complexes) | Cohen's d (separation of binder and decoy ipTM score distributions over all 22,494 complexes), unitless | 0.2 | 0.75 | Protenix |
| POISK epitope contact recovery | POISK binder records with mappable patent epitope annotations (1,232) | Success rate (at least one annotated epitope residue with a heavy atom within 4 angstrom of an antibody heavy atom, over 1,232 binder records with mappable patent epitope annotations), percent | 55.4 | 62.8 | Protenix |
| POISK pairwise binder-versus-decoy accuracy at 10% coverage | POISK co-folding binder-decoy benchmark set (22,494 complexes) | Paired accuracy (binder scored above its matched putative decoy by full-Fv ipSAE, top 10% of the 11,247 pairs ranked by within-pair maximum ipSAE, exact ties at half credit), fraction | 0.641 | 0.978 | Protenix |
| POISK pairwise binder-versus-decoy accuracy at 100% coverage | POISK co-folding binder-decoy benchmark set (22,494 complexes) | Paired accuracy (binder scored above its matched putative decoy by full-Fv ipSAE, 100% coverage of the 11,247 pairs, exact ties at half credit), fraction | 0.581 | 0.719 | Protenix |
| POISK pairwise binder-versus-decoy accuracy at 25% coverage | POISK co-folding binder-decoy benchmark set (22,494 complexes) | Paired accuracy (binder scored above its matched putative decoy by full-Fv ipSAE, top 25% of the 11,247 pairs ranked by within-pair maximum ipSAE, exact ties at half credit), fraction | 0.622 | 0.948 | Protenix |
| POISK pairwise binder-versus-decoy accuracy at 50% coverage | POISK co-folding binder-decoy benchmark set (22,494 complexes) | Paired accuracy (binder scored above its matched putative decoy by full-Fv ipSAE, top 50% of the 11,247 pairs ranked by within-pair maximum ipSAE, exact ties at half credit), fraction | 0.602 | 0.87 | Protenix |
| POISK pairwise binder-versus-decoy accuracy at 75% coverage | POISK co-folding binder-decoy benchmark set (22,494 complexes) | Paired accuracy (binder scored above its matched putative decoy by full-Fv ipSAE, top 75% of the 11,247 pairs ranked by within-pair maximum ipSAE, exact ties at half credit), fraction | 0.591 | 0.781 | Protenix |
| POISK 1-in-50 screening simulation, top-1 retrieval | POISK 1-in-50 screening targets (27 antigen constructs) | Top-1 accuracy (retrieval; one patent-supported binder ranked against 49 putative decoys for the same antigen by full-Fv ipSAE, mean over 27 antigen constructs, 20 trials per construct per seed, 25 seeds), fraction | 0.03 | 0.29 | Protenix |
| POISK 1-in-50 screening simulation, top-1 retrieval | POISK 1-in-50 screening targets (27 antigen constructs) | Top-1 accuracy (retrieval; one patent-supported binder ranked against 49 putative decoys for the same antigen by full-Fv ipSAE, mean over 27 antigen constructs, 20 trials per construct per seed, 25 seeds), percent | 3.1 | 28.9 | Protenix |
What disagrees
- studies
- 2
Follow-up
Current state: Not yet reviewed.
No one has reviewed this finding yet.
How to help resolve this
If you know why these studies disagree, for example a different population, metric or preprocessing step, or you have a corrected value, tell us through the contribution form. A reviewer checks every submission against its source before anything changes, and the step is added to the follow-up above.
Finding discrepancy-1593d9dbfee36f07 · All contested results · How findings are made and followed up