rewire.itbenchmarks
Configuration

TAPE Transformer (ATOM3D baseline)

The TAPE Transformer learns contextual protein representations using masked-residue pretraining and a task-specific prediction head.

Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3

2 evaluations · 2 results

How it worksTAPE Transformer workflow
TAPE Transformer workflow1. Protein sequence. Then: 2. Residue and position embeddings. Then: 3. Transformer encoder. Then: 4. Task headTAPE Transformer workflow1. Protein sequence. Then: 2. Residue and position embeddings. Then: 3. Transformer encoder. Then: 4. Task headTAPE Transformer workflow1. Protein sequence. Then: 2. Residue and position embeddings. Then: 3. Transformer encoder. Then: 4. Task head

Conceptual summary of the documented data flow; optional inputs and configured downstream stages must be reported for a reproducible evaluation.

Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3

Overview

Model type

BERT-style protein transformer encoder

Inputs

Amino-acid sequence in the tokenizer expected by the selected implementation.

Outputs

Residue/sequence representations and predictions from the chosen task head.

Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3

limited source coverage · Automated source review, 2026-09-16. All specifications and missing details

Evaluations and results

2 evaluations · 2 results. Different protocols are not a single leaderboard.

Filter evaluations

Applied filters: All linked evaluations

Exact evaluated configurations and original reported results
Tested configurationProtocol and datasetFindingEvidence and details
Configuration: TAPE Transformer (ATOM3D baseline) (cited as [Rao et al., 2019])Task: ATOM3D MSP: Mutation stability prediction
Dataset subset: ATOM3D MSP (ATOM3D split)
0.554 auroc
fraction · higher

Uncertainty: Not reported

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

TAPE Transformer (ATOM3D baseline) on ATOM3D MSP: Mutation stability prediction

Trained and scored under the ATOM3D split for this task. Asterisks in the paper mark a run whose training data differed.

Aggregation: Not reported

ATOM3D: Tasks On Molecules in Three Dimensions · Table 4, row(MSP AUROC), column([Rao et al., 2019])
Configuration: TAPE Transformer (ATOM3D baseline) (cited as [Rao et al., 2019])Task: ATOM3D RES: Residue identity
Dataset subset: ATOM3D RES (ATOM3D split)
0.3 accuracy
fraction · higher

Uncertainty: Not reported

Coverage: Not reported scored / Not reported eligible

Author-reported evaluation · Source checked
Methods, coverage and source

TAPE Transformer (ATOM3D baseline) on ATOM3D RES: Residue identity

Trained and scored under the ATOM3D split for this task. Asterisks in the paper mark a run whose training data differed.

Aggregation: Not reported

ATOM3D: Tasks On Molecules in Three Dimensions · Table 4, row(RES accuracy), column([Rao et al., 2019])

Source checking is not independent reproduction. Release 2026-09-29-06401fd5b220.

Use this model

How it works, versions and access

Related profile: TAPE Transformer. This page retains the exact record and its evaluation context.

This configuration

TAPE, a transformer over protein sequence, as the ATOM3D text describes it.

record
TAPE Transformer (ATOM3D baseline)
configuration
Not reported
entity type
Configuration

How it works

How it works

The TAPE Transformer learns contextual protein representations using masked-residue pretraining and a task-specific prediction head. The June 2019 preprint uses 12 transformer layers, width 512 and eight heads (38M parameters). The later PyTorch BertConfig defaults to width 768 and 12 heads; these are different configurations. The documented inputs are amino-acid sequence in the tokenizer expected by the selected implementation. The output consists of residue/sequence representations and predictions from the chosen task head.

Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
Versions and reproducibility

June 2019 TAPE preprint configurations and the later PyTorch package are distinct; the current README warns it is not an exact reproduction. The inspected default BertConfig sets max_position_embeddings to 8,096; this is an implementation default, not evidence that a historical TAPE checkpoint was trained at that length.

Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
Strengths, limitations and unresolved questions

Strengths and limitations

Strengths and considerations

Limitations and conditions

Profile review details

Inspected pinned official documentation, relevant implementation files and named primary-paper sections. Claims are limited to those artifacts. Remaining field extraction and identity conflicts are explicit; no new performance claims, model runs or human review are implied.

Stable record: discovery-model-tape-transformer

Specifications

Inputs, training, access and other details

Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Inputs, outputs and configuration
PropertyDescription and evidence
Model typeBERT-style protein transformer encoder
Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
ArchitectureThe June 2019 preprint uses 12 transformer layers, width 512 and eight heads (38M parameters). The later PyTorch BertConfig defaults to width 768 and 12 heads; these are different configurations.
Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
InputsAmino-acid sequence in the tokenizer expected by the selected implementation.
Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
OutputsResidue/sequence representations and predictions from the chosen task head.
Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
Parameters38M for the June 2019 preprint Transformer; do not apply that total to the later PyTorch defaults or every task head.
Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
Known versionsJune 2019 TAPE preprint configurations and the later PyTorch package are distinct; the current README warns it is not an exact reproduction.
Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
Training dataThe June 2019 TAPE preprint uses 31M Pfam domains, with held-out families and a separate random split; downstream task heads are fitted on their own task data.
Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
Training cutoffThe June 2019 TAPE preprint identifies the Pfam-domain corpus and split procedure but does not state one latest-sequence deposition date. Later package defaults are a different implementation. · Not reported in inspected sources
Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
Context limitsThe inspected default BertConfig sets max_position_embeddings to 8,096; this is an implementation default, not evidence that a historical TAPE checkpoint was trained at that length.
Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
Weights licenceSeparate checkpoint-distribution terms are not stated in the inspected release documentation and licence material. The source-code licence alone is not recorded as an explicit weight grant. · Not reported in inspected sources
Sources (8)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF; songlab-cal/tape: LICENSE · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3; LICENSE: licence text
AccessOfficial project documentation and implementation: https://github.com/songlab-cal/tape
Sources (7)songlab-cal/tape: README.md; songlab-cal/tape: tape/models/modeling_lstm.py; songlab-cal/tape: tape/models/modeling_resnet.py; songlab-cal/tape: tape/models/modeling_unirep.py; songlab-cal/tape: tape/models/modeling_onehot.py; songlab-cal/tape: tape/models/modeling_bert.py; tape: Primary paper PDF · tape/models/modeling_bert.py: configuration and model classes; README.md: opening maintenance warning, Examples and Data; June 2019 TAPE preprint Sections 4.1,5 and AppendixA.3
Code licenceBSD-3-Clause
Sourcessonglab-cal/tape: LICENSE · LICENSE: licence text

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

2 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-29-06401fd5b220
Property and statementOriginal source and locationReview and provenance
Relationship: family
discovery-model-tape-transformer
Individual claims
ATOM3D: Tasks On Molecules in Three Dimensions

Original source ↗

ATOM3D §5.1, p.7; App. F.3 and F.4, p.24. TAPE arXiv:1906.08230v1 Table 1, p.6.

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: Primary full-text snapshot retrieved 2026-09-17; exact bytes pinned by SHA-256
Retrieved: 2026-09-17T08:06:28.182997+00:00

source checked

automated source review · 2026-09-24

Audit details

Source review establishes this relationship only. Exact evaluated configurations and original numerical review status remain unchanged. ATOM3D identifies the compared model as the TAPE transformer. Family attribution only: RES reuses a TAPE paper value and MSP uses an ATOM3D-modified head.

Field: links:family:discovery-model-tape-transformer

Claim: model-evaluation-identity-1b29ffb764d95d1a3be2

Source artifact SHA-256: 92656c20a15311c32bed9edc7f465bb26eb30338f40324fe43bec4b1fc6a7890

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Relationship: family
discovery-model-tape-transformer
Individual claims
Evaluating Protein Transfer Learning with TAPE

Original source ↗

ATOM3D §5.1, p.7; App. F.3 and F.4, p.24. TAPE arXiv:1906.08230v1 Table 1, p.6.

Shared locator for this statement’s cited sources; not a separate locator for each citation.

Version: 1906.08230v1
Retrieved: 2026-09-16T20:23:48.238777+00:00

source checked

automated source review · 2026-09-24

Audit details

Source review establishes this relationship only. Exact evaluated configurations and original numerical review status remain unchanged. ATOM3D identifies the compared model as the TAPE transformer. Family attribution only: RES reuses a TAPE paper value and MSP uses an ATOM3D-modified head.

Field: links:family:discovery-model-tape-transformer

Claim: model-evaluation-identity-1b29ffb764d95d1a3be2

Source artifact SHA-256: 6ee0c3e6e870635cba8fa67e0a4abc5598c0ab2a10ba127a46b67ac450ae0168

Hash scope: Exact retrieved primary paper artifact bytes.

Inspected artifact

Sources and history

View linked audit checks and correction history

Release 2026-09-29-06401fd5b220 · Record review: source checked

2 source records and release historyDownload this release
Technical metadata and extraction receipts

Stable ID: atom3d-method-rao-et-al-2019

areas
molecular-interactions
source locator
Table 4, column([Rao et al., 2019])
missing metadata
checkpoint revision: unreported; parameters: unextracted
source label
[Rao et al., 2019]
source identity
status: resolved; label form: bracketed_citation; display name: TAPE Transformer (ATOM3D baseline); identity: TAPE Transformer; configuration: RES uses the accuracy reported in the TAPE paper. MSP uses the public TAPE implementation with a modified head, trained by the ATOM3D authors.; basis: ATOM3D §5.1 describes the comparison as 'the TAPE model [Rao et al., 2019], a transformer architecture that operates on protein sequence', and App. F.3 names 'the transformer architecture TAPE'. The RES value 0.30 also equals the Transformer held-out-family accuracy in TAPE Table 1. That match corroborates the text; it is not the basis of the identity.; source ids: evidence-expansion-atom3d-92656c20; evidence-expansion-evidence-discovery-final-tape-6ee0c3e6; source locator: ATOM3D arXiv:2012.04035v4 §5.1 and Table 4, p.7; App. F.3 and F.4 with footnote 9, p.24; Table 8 RES and MSP rows, p.28. TAPE arXiv:1906.08230v1 Table 1 and Architectures and Training, p.6.; known details: label: RES value; value: Taken from TAPE's reported language-modelling accuracy on held-out Pfam families, which ATOM3D treats as a sequence-only counterpart of RES. Table 8 marks it as trained on different data.; source ids: evidence-expansion-atom3d-92656c20; evidence-expansion-evidence-discovery-final-tape-6ee0c3e6; source locator: ATOM3D App. F.3, p.24; Table 8, p.28; TAPE Table 1, p.6; label: MSP; value: Public TAPE implementation with its sequence-to-sequence head modified to predict the effect of mutations at given positions.; source ids: evidence-expansion-atom3d-92656c20; source locator: ATOM3D App. F.4, p.24; unknown: label: MSP initial weights; note: ATOM3D does not say whether MSP training started from pretrained TAPE weights.; label: Checkpoint; note: No checkpoint or implementation revision is identified.; original name: [Rao et al., 2019]; original description: TAPE, a transformer over protein sequence, as the ATOM3D text describes it.; review: method: automated_source_review; date: 2026-09-24; note: AI-assisted review against the cited primary sources. No human scientific review. Values, locators and comparison conditions are unchanged.
Related records

Suggest a correction