| Configuration: Bacformer genome embeddings for strain clustering (checkpoint not resolved) | Protocol: BacBench strain clustering, adjusted Rand index Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.857 ± 0.035 Adjusted rand index (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.035 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceBacformer on BacBench strain clustering, adjusted Rand index Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'Bacformer', column 'ARI' |
|---|
| Configuration: Bacformer genome embeddings for strain clustering (checkpoint not resolved) | Protocol: BacBench strain clustering, average silhouette width Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.250 ± 0.022 Silhouette score (average silhouette width of the Leiden clustering of mean-pooled genome embeddings, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.022 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceBacformer on BacBench strain clustering, average silhouette width Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'Bacformer', column 'ASW' |
|---|
| Configuration: Bacformer Large genome embeddings for strain clustering (checkpoint not resolved) | Protocol: BacBench strain clustering, adjusted Rand index Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.970 ± 0.017 Adjusted rand index (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.017 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceBacformer Large on BacBench strain clustering, adjusted Rand index Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'Bacformer Large', column 'ARI' |
|---|
| Configuration: Bacformer Large genome embeddings for strain clustering (checkpoint not resolved) | Protocol: BacBench strain clustering, average silhouette width Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.460 ± 0.031 Silhouette score (average silhouette width of the Leiden clustering of mean-pooled genome embeddings, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.031 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceBacformer Large on BacBench strain clustering, average silhouette width Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'Bacformer Large', column 'ASW' |
|---|
| Configuration: Bacformer Large genome embeddings for strain clustering (checkpoint not resolved) | Protocol: BacBench strain clustering, normalised mutual information Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.985 ± 0.006 Normalized mutual information (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.006 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceBacformer Large on BacBench strain clustering, normalised mutual information Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'Bacformer Large', column 'NMI' |
|---|
| Configuration: Bacformer genome embeddings for strain clustering (checkpoint not resolved) | Protocol: BacBench strain clustering, normalised mutual information Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.924 ± 0.015 Normalized mutual information (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.015 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceBacformer on BacBench strain clustering, normalised mutual information Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'Bacformer', column 'NMI' |
|---|
| Configuration: ESM-2 (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, adjusted Rand index Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.520 ± 0.028 Adjusted rand index (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.028 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceESM-2 on BacBench strain clustering, adjusted Rand index Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'ESM-2', column 'ARI' |
|---|
| Configuration: ESM-2 (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, average silhouette width Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.191 ± 0.022 Silhouette score (average silhouette width of the Leiden clustering of mean-pooled genome embeddings, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.022 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceESM-2 on BacBench strain clustering, average silhouette width Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'ESM-2', column 'ASW' |
|---|
| Configuration: ESM-2 (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, normalised mutual information Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.732 ± 0.007 Normalized mutual information (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.007 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceESM-2 on BacBench strain clustering, normalised mutual information Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'ESM-2', column 'NMI' |
|---|
| Configuration: ESM-C (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, adjusted Rand index Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.707 ± 0.022 Adjusted rand index (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.022 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceESM-C on BacBench strain clustering, adjusted Rand index Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'ESM-C', column 'ARI' |
|---|
| Configuration: ESM-C (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, average silhouette width Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.277 ± 0.031 Silhouette score (average silhouette width of the Leiden clustering of mean-pooled genome embeddings, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.031 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceESM-C on BacBench strain clustering, average silhouette width Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'ESM-C', column 'ASW' |
|---|
| Configuration: ESM-C (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, normalised mutual information Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.861 ± 0.009 Normalized mutual information (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.009 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceESM-C on BacBench strain clustering, normalised mutual information Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'ESM-C', column 'NMI' |
|---|
| Configuration: gLM2 (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, adjusted Rand index Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.710 ± 0.028 Adjusted rand index (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.028 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourcegLM2 on BacBench strain clustering, adjusted Rand index Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'gLM2', column 'ARI' |
|---|
| Configuration: gLM2 (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, average silhouette width Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.381 ± 0.012 Silhouette score (average silhouette width of the Leiden clustering of mean-pooled genome embeddings, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.012 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourcegLM2 on BacBench strain clustering, average silhouette width Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'gLM2', column 'ASW' |
|---|
| Configuration: gLM2 (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, normalised mutual information Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.836 ± 0.009 Normalized mutual information (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.009 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourcegLM2 on BacBench strain clustering, normalised mutual information Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'gLM2', column 'NMI' |
|---|
| Configuration: ProtBERT (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, adjusted Rand index Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.720 ± 0.028 Adjusted rand index (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.028 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceProtBERT on BacBench strain clustering, adjusted Rand index Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'ProtBERT', column 'ARI' |
|---|
| Configuration: ProtBERT (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, average silhouette width Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.326 ± 0.037 Silhouette score (average silhouette width of the Leiden clustering of mean-pooled genome embeddings, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.037 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceProtBERT on BacBench strain clustering, average silhouette width Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'ProtBERT', column 'ASW' |
|---|
| Configuration: ProtBERT (frozen embeddings, no trained head) | Protocol: BacBench strain clustering, normalised mutual information Dataset: BacBench strain clustering dataset of 6,071 MGnify MAGs | 0.866 ± 0.012 Normalized mutual information (Leiden clustering of mean-pooled genome embeddings scored against species labels, at the resolution and neighbour count that maximise the mean of the three reported metrics) unitless · higher Uncertainty: SD 0.012 Coverage: Not reported scored / Not reported eligible | Author-reported evaluation · Source checkedMethods, coverage and sourceProtBERT on BacBench strain clustering, normalised mutual information Final-layer hidden states are mean-pooled into one vector per genome and clustered with Leiden over a grid of resolutions 0.1, 0.25, 0.5 and 1.0 and neighbour counts 5, 10 and 15; the combination that maximises the mean of the adjusted Rand index, normalised mutual information and average silhouette width at the species level is retained. Clustering is run with Scanpy. Aggregation: Not reported Bacformer preprint, bioRxiv 2025.07.20.665723 version 3 · Table S12, row 'ProtBERT', column 'NMI' |
|---|