Benchmark

FLIP

FLIP evaluates protein-sequence representations using multiple deliberately defined train/test splits.

Sources

J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits

155 evaluations · 155 results

Overview

Datasets

Protein sequence/property collections distributed as raw data, processed splits and FASTA resources.

Sources

J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits

Metrics

Spearman rank correlation between predicted and measured fitness on each landscape/split.

Sources

flip primary benchmark evidence · Sections 3–5; Tables 2 and 4–7

Allowed inputs

Protein sequences with property labels.

Sources

J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits

Evaluation procedure diagram

How it worksEvaluation procedure

Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions.

Sources (2)

J-SNACKKB/FLIP official source; flip primary benchmark evidence · Pinned README: repository organization; Splits; Sections 3–5; Tables 2 and 4–7

limited source coverage · Automated source review, 2026-09-16. All specifications and missing details

Results

Each comparison retains its reviewed evaluation scope, dataset and metric. Results are shown without a pooled ranking.

Find a protocol, dataset or metric

Evaluation scope and datasetMetric and reported setup

FLIP AAV-1-VS-REST: AAV fitness prediction, 1-vs-rest split

spearman (correlation) · Higher values are better.

FLIP AAV-1-VS-REST: AAV fitness prediction, 1-vs-rest split · FLIP AAV, 1-vs-rest split (FLIP split)

Evidence origin: Author-reported evaluation.

flip primary benchmark evidence · Table 5, column(1-vs-rest)

The split is the task here. Figures from different splits of the same landscape are not comparable.
The random sampled splits in Table 7 are the authors' own illustration of an optimistic evaluation, not a headline result.
NA marks a baseline that cannot be applied to that landscape, and is recorded as no result rather than as a zero.

Comparison details and limitations

Every method FLIP reports on AAV fitness prediction, 1-vs-rest split, scored with Spearman correlation on FLIP AAV, 1-vs-rest split.

Author-reported numbers, source checked but not independently reproduced.

Automated source review: 2026-09-18. Numerical source review does not establish independent reproduction.

Dots show point estimates. Whiskers show only explicitly defined uncertainty (standard deviation, standard error or a labelled interval); their definitions remain in Table. Unresolved uncertainty is not plotted. Differences do not establish statistical significance.

Find a tested model or method

Showing 12 of 12 matching rows.

Zoom to observed range

Tested configuration

-1-0.500.51

Reported score

Methods and evaluation design

Procedure, tasks and evaluated configurations

How it works

Evaluation methodology

FLIP evaluates sequence-to-fitness models under protein-engineering distribution shifts. It separates landscapes from their partitions: the same assay can test random interpolation, higher mutation counts, higher fitness or transfer across sequence groups. Comparisons are meaningful within the same landscape and split.

Sources

flip primary benchmark evidence · Sections 3–5; Tables 2 and 4–7

Evaluation design

Benchmarks bring together tasks and protocols. A task describes the biological question; a protocol defines a particular test.

Tasks

These source-backed links do not make different protocols or scores interchangeable.

Baseline coverage

Reference methods help show what a model adds beyond simple controls. We track a null control and a conventional method for each protocol.

No concrete protocols are explicitly linked to this suite. Protocol identification and baseline selection are outstanding.

Protocol coverage CSV · Model evaluation matrix · Source table · Release and checksums

Coverage is derived from release 2026-09-29-06401fd5b220. Source citations describe the original records; they do not validate an unreviewed baseline proposal. No results have been generated by this audit.

Run this benchmark

The repository points to baseline implementations, dataset downloads and split definitions. It explicitly marks some splits cautionary or obsolete; only the chosen active split can support reported performance. The top-level README does not give a complete baseline command.

A maintained rewire runner has not been verified for this benchmark. Check data access, weights, licences, dependencies and hardware in the linked official documentation; requirements have not been fully extracted.

J-SNACKKB/FLIP / README.md · README.md lines 6–26 (Folder breakup and split semaphore)

Strengths, limitations and unresolved questions

Strengths and limitations

Strengths and considerations

Split semaphores explicitly mark active, cautionary and obsolete comparison settings.
Sources
J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits

Limitations and conditions

Random and extrapolative splits answer different questions. Closely related mutants are intentional in engineering tasks, so independence cannot be reduced to a universal sequence-identity threshold.
Sources
flip primary benchmark evidence · Sections 3–5; Tables 2 and 4–7

Profile review details

Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction.

Stable record: discovery-benchmark-flip

Specifications

Inputs, training, access and other details

Explanatory profile: limited source coverage · Automated source review, 2026-09-16. Review applies to the cited claims; unresolved fields are listed below. Numerical results retain their own review status.

Data, procedure and scoring
Property	Description and evidence
Datasets	Protein sequence/property collections distributed as raw data, processed splits and FASTA resources. Sources J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits
Splits	The splits directory documents biological/statistical split logic; multiple splits may exist for one dataset. Sources J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits
Metrics	Spearman rank correlation between predicted and measured fitness on each landscape/split. Sources flip primary benchmark evidence · Sections 3–5; Tables 2 and 4–7
Baselines	A baselines directory provides reference implementations. Sources J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits
Leakage controls	FLIP contrasts random partitions with mutation-number, fitness and sequence-family or diversity partitions. Its purpose is to expose distribution shifts; a random partition is an easier control, not an equivalent test. Sources flip primary benchmark evidence · Sections 3–5; Tables 2 and 4–7
Uncertainty	The inspected baseline tables report point correlations. Their Methods do not define a common repeated-seed or bootstrap confidence interval for all landscape/split results. · Not reported in inspected sources Sources flip primary benchmark evidence · Sections 3–5; Tables 2 and 4–7
Entity type	Protein sequence learning benchmark with multiple split regimes. Sources J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits
Organisms	GB1 binding variants, adeno-associated virus capsid variants and Meltome proteins across the tree of life; the landscapes have different organism and assay scopes. Sources flip primary benchmark evidence · Sections 3–5; Tables 2 and 4–7
Assays	Measured protein properties from the selected source datasets. Sources J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits
Allowed inputs	Protein sequences with property labels. Sources J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits
Adaptation	Supervised learning on the chosen training partition; different splits test different generalization conditions. Sources J-SNACKKB/FLIP official source · Pinned README: repository organization; Splits

Evidence

Source checking verifies the cited claim or transcription. It does not establish independent reproduction.

Papers and result coverage

Last literature check: 2026-09-17. Primary-paper discovery and source inspection. Source-checked results are not independently reproduced experiments.

Paper or primary resource	Version	Reference
flip primary benchmark evidence	2021 manuscript	Read source

Historical gaps recorded on 2026-09-17

The catalogue now holds 155 result rows for this benchmark. A note below about pending extraction describes the state on 2026-09-17 and may since have been answered by a later batch. The result rows and their sources are the current record.

No suite-wide raw score. Random splits and biologically motivated splits are distinct.
Thermostability NA entries are not zeros; Table 4 includes negative Spearman correlations.
No new numeric extraction in this scoped release; full tables identified.

Search and extraction details

primary protocol screened

Searches

GlycanML benchmark 2405.16206
AMBER metagenome binning assessment 2018 PMC6022608
BEELINE gene regulatory network benchmark 2020 Pratapa
FLIP benchmark protein fitness landscape inference 2021

Evidence locations

Tables 2–4 and 6
Sections 4–6

Evidence table

Inspect claims, sources and review details

Trace each statement to its source and review. A context-only reference supports the record generally; it does not verify an individual field. Source checking does not reproduce an experiment.

One row per statement and cited source. Multiple citations are not independent evaluations. Shared locators are labelled explicitly.

21 evidence rows matching the loaded filters

Claims, original sources and review scope · Release 2026-09-29-06401fd5b220
Property and statement	Original source and location	Review and provenance
Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims	flip primary benchmark evidence Original source ↗ Pinned README: repository organization; Splits; Sections 3–5; Tables 2 and 4–7 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 2021 manuscript Retrieved: 2026-09-16T21:07:13.889041+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.caption` Source artifact SHA-256: `afcf360c88a7a4ae153b3c2d8d4fa6d4ac0abe84f2131b94409abff6447ce363` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Diagram caption Conceptual procedure. Task variants and protocol versions retain their separate scoring conditions. Individual claims	J-SNACKKB/FLIP official source Original source ↗ Pinned README: repository organization; Splits; Sections 3–5; Tables 2 and 4–7 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 62cace8735f5610e2743cf06ce0f944b37fffaa6 Retrieved: 2026-09-16T10:30:21.863141+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.caption` Source artifact SHA-256: `f6e3b46a5f6744806846ccb4a054bcf3bec3da3d4249ada42fb9acac2a32024d` Hash scope: Hash scope not separately documented; inspect source record
Diagram steps Allowed inputs: Protein sequences with property labels. Splits: The splits directory documents biological/statistical split logic; multiple splits may exist for one dataset. Metrics: Spearman rank correlation between predicted and measured fitness on each landscape/split. Individual claims	flip primary benchmark evidence Original source ↗ Pinned README: repository organization; Splits; Sections 3–5; Tables 2 and 4–7 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 2021 manuscript Retrieved: 2026-09-16T21:07:13.889041+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.steps` Source artifact SHA-256: `afcf360c88a7a4ae153b3c2d8d4fa6d4ac0abe84f2131b94409abff6447ce363` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Diagram steps Allowed inputs: Protein sequences with property labels. Splits: The splits directory documents biological/statistical split logic; multiple splits may exist for one dataset. Metrics: Spearman rank correlation between predicted and measured fitness on each landscape/split. Individual claims	J-SNACKKB/FLIP official source Original source ↗ Pinned README: repository organization; Splits; Sections 3–5; Tables 2 and 4–7 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 62cace8735f5610e2743cf06ce0f944b37fffaa6 Retrieved: 2026-09-16T10:30:21.863141+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.steps` Source artifact SHA-256: `f6e3b46a5f6744806846ccb4a054bcf3bec3da3d4249ada42fb9acac2a32024d` Hash scope: Hash scope not separately documented; inspect source record
Diagram title Evaluation procedure Individual claims	flip primary benchmark evidence Original source ↗ Pinned README: repository organization; Splits; Sections 3–5; Tables 2 and 4–7 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 2021 manuscript Retrieved: 2026-09-16T21:07:13.889041+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.title` Source artifact SHA-256: `afcf360c88a7a4ae153b3c2d8d4fa6d4ac0abe84f2131b94409abff6447ce363` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact
Diagram title Evaluation procedure Individual claims	J-SNACKKB/FLIP official source Original source ↗ Pinned README: repository organization; Splits; Sections 3–5; Tables 2 and 4–7 Shared locator for this statement’s cited sources; not a separate locator for each citation. Version: 62cace8735f5610e2743cf06ce0f944b37fffaa6 Retrieved: 2026-09-16T10:30:21.863141+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.diagram.title` Source artifact SHA-256: `f6e3b46a5f6744806846ccb4a054bcf3bec3da3d4249ada42fb9acac2a32024d` Hash scope: Hash scope not separately documented; inspect source record
Datasets Protein sequence/property collections distributed as raw data, processed splits and FASTA resources. Individual claims	J-SNACKKB/FLIP official source Original source ↗ Pinned README: repository organization; Splits Version: 62cace8735f5610e2743cf06ce0f944b37fffaa6 Retrieved: 2026-09-16T10:30:21.863141+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.0.value` Source artifact SHA-256: `f6e3b46a5f6744806846ccb4a054bcf3bec3da3d4249ada42fb9acac2a32024d` Hash scope: Hash scope not separately documented; inspect source record
Splits The splits directory documents biological/statistical split logic; multiple splits may exist for one dataset. Individual claims	J-SNACKKB/FLIP official source Original source ↗ Pinned README: repository organization; Splits Version: 62cace8735f5610e2743cf06ce0f944b37fffaa6 Retrieved: 2026-09-16T10:30:21.863141+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.1.value` Source artifact SHA-256: `f6e3b46a5f6744806846ccb4a054bcf3bec3da3d4249ada42fb9acac2a32024d` Hash scope: Hash scope not separately documented; inspect source record
Adaptation Supervised learning on the chosen training partition; different splits test different generalization conditions. Individual claims	J-SNACKKB/FLIP official source Original source ↗ Pinned README: repository organization; Splits Version: 62cace8735f5610e2743cf06ce0f944b37fffaa6 Retrieved: 2026-09-16T10:30:21.863141+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.10.value` Source artifact SHA-256: `f6e3b46a5f6744806846ccb4a054bcf3bec3da3d4249ada42fb9acac2a32024d` Hash scope: Hash scope not separately documented; inspect source record
Metrics Spearman rank correlation between predicted and measured fitness on each landscape/split. Individual claims	flip primary benchmark evidence Original source ↗ Sections 3–5; Tables 2 and 4–7 Version: 2021 manuscript Retrieved: 2026-09-16T21:07:13.889041+00:00	source checked automated source review · 2026-09-16 Audit details Primary paper and/or task implementation reviewed for the explicitly cited methodology claims. Scope-limited absence is recorded only after the documented source search; no model runs or independent reproduction. Field: `attributes.profile.facts.2.value` Source artifact SHA-256: `afcf360c88a7a4ae153b3c2d8d4fa6d4ac0abe84f2131b94409abff6447ce363` Hash scope: Hash scope not separately documented; inspect source record Inspected artifact

All evidence (CSV)All evidence (JSONL)How to read the evidence

Sources and history

View linked audit checks and correction history

Release 2026-09-29-06401fd5b220 · Record review: discovered

4 source records and release history

J-SNACKKB/FLIP official source · Original source · 62cace8735f5610e2743cf06ce0f944b37fffaa6
flip primary benchmark evidence · Original source · 2021 manuscript
J-SNACKKB/FLIP / README.md · Original source · Git commit 62cace8735f5610e2743cf06ce0f944b37fffaa6
flip primary benchmark evidence · Original source · 2021 manuscript

Download this release

Technical metadata and extraction receipts

Stable ID: discovery-benchmark-flip

areas: protein-function
entity level: suite
scope note: Specialist molecular or omics evaluation; protocol details require review before numerical comparison.
task: Generalisation in protein fitness landscapes
version: Not reported
benchmark research: review date: 2026-09-17; status: primary_protocol_screened; primary sources: expansion-p3-flip; inspected locators: Tables 2–4 and 6; Sections 4–6; searched queries: GlycanML benchmark 2405.16206; AMBER metagenome binning assessment 2018 PMC6022608; BEELINE gene regulatory network benchmark 2020 Pratapa; FLIP benchmark protein fitness landscape inference 2021; gaps: No suite-wide raw score. Random splits and biologically motivated splits are distinct.; Thermostability NA entries are not zeros; Table 4 includes negative Spearman correlations.; No new numeric extraction in this scoped release; full tables identified.; claim scope: Primary-paper discovery and source inspection. Source-checked results are not independently reproduced experiments.
historical missing metadata: dataset release: unextracted; metric implementation: unextracted; split manifest: unextracted; version: unextracted
metadata review scope: historical_missing_metadata preserves the original discovery state. Current descriptive evidence and missingness are recorded in profile.facts; numerical-result review is separate.
entity classification: review date: 2026-09-17; rationale: The cited profile describes a collection of evaluation tasks or protocols; retain it as the top-level benchmark suite. Its datasets and individual protocols remain separate records.; source ids: src-discovery-j-snackkb-flip; source locator: Pinned README: repository organization; Splits; ambiguities: None recorded
run documentation: record id: discovery-benchmark-flip; source ids: run-doc-flip-readme-md-62cace87; status: official_documentation_linked; summary: The repository points to baseline implementations, dataset downloads and split definitions. It explicitly marks some splits cautionary or obsolete; only the chosen active split can support reported performance. The top-level README does not give a complete baseline command.; source locator: README.md lines 6–26 (Folder breakup and split semaphore)

Related records

Suggest a correction