Biological model benchmark database

67,462 records across 17 record types. These counts describe catalogue coverage, not model performance.

Benchmark evidence coverage

This is a dated collection, not an exhaustive model census. Models, methods, evaluated configurations and pipelines are listed separately. Counts describe records, not unique checkpoints or independent experiments.

{
  "entity_schema_version": "1.1",
  "entity_migration": {
    "baseline_release": "2026-09-17-5054ddf2a281",
    "note": "Separate models, methods, configurations, pipelines, hosted services, benchmarks, tasks, protocols and evaluators. IDs, printed scores and archived releases remain unchanged."
  },
  "acquisition_review": {
    "date": "2026-09-19",
    "scope": "Nine benchmark result collections; PEtab timing data remains quarantined. Source cells checked independently; not experimental reproduction."
  },
  "run_instructions": {
    "review": "Official source instructions; not executed by rewire",
    "guides": 7,
    "documentation_audits": 39
  },
  "run_recipe_coverage": {
    "recipes": 30,
    "evaluations_with_verified_recipe_links": 9,
    "top_level_benchmarks": 82,
    "official_documentation_or_gap": 36,
    "note": "Source-reviewed instructions and exact applicability links; execution receipts do not establish reproduction of published scores. Production contributions remain disabled."
  },
  "research_generator_sha256": "14d01fcbe7cce144376eeb0be5dc527a5a4ec615ba1c039134eebad74ed56beb",
  "evidence_table_generator_sha256": "53e835a4b212bd058c7eef3215a72e4ddfd8f90d47a813f495b28d11c03b39e9",
  "profile_coverage": {
    "total": 694,
    "fact_status_counts": {
      "source_checked": 3878,
      "unreported": 702,
      "unextracted": 49,
      "unavailable": 0,
      "inapplicable": 247,
      "unclassified": 0
    },
    "note": "Source review applies to individual cited claims. Missing fields and inaccessible evidence remain explicit; profile coverage is not independent experimental verification.",
    "reviewed": 92,
    "limited": 602
  },
  "benchmark_paper_review": {
    "date": "2026-10-12",
    "top_level_benchmarks": 82,
    "benchmark_pages_with_results": 54,
    "benchmark_pages_with_figures": 38,
    "published_comparison_figures": 749,
    "source_checked_result_rows": 38597,
    "scope": "Counts derived from this release through the production relationship and chart gates. Figures are source-specific, and metric rows are not independent experiments. Source review is not reproduction."
  },
  "changelog": [
    "Audit all 17 use cases; add source-reviewed, protocol-specific literature measurements and conventional baselines, with explicit remaining decision and transfer gaps. Automated source review is not independent experimental reproduction or human scientific review.",
    "Add two sourced research use cases linking questions to exact existing evaluations, with separate MFASS and AMFR protocol scopes, explicit clinical evidence gaps and an independently versioned, release-pinned use-case artifact. Add two documentation source records; preserve every prior scientific record and numerical result.",
    "Add four MFASS matched canonical-annotation configurations and 16 exact metric rows on the identical 8,297/8,324 scored subset; retain 23 assembly-orientation exclusions and four canonical transcript-scope exclusions, with pinned automated evidence and separate comparison panels.",
    "Add complete bounded primary-source result tables for BEELINE, CAFA, CAMI, CAPRI, CASP, FLIP2, PLINDER, scIB and provisional Virtual Cell Challenge 2026 validation.",
    "Preserve PEtab timing candidates and four conflicting FLIP2 values in acquisition staging with explicit limitations.",
    "Add append-only linked audit runs, field checks, source retrieval receipts and correction history; unresolved fields remain explicit.",
    "Recover 16 omitted ProteinBench percentage results and correct percent units; original printed and numeric values remain unchanged.",
    "Correct DART-Eval correlation units and remove unsupported standard-deviation labels from TDC printed plus/minus spreads.",
    "Recheck pinned extraction inputs, reject unrecognised cells and authenticate PDF-to-text transformations.",
    "Add complete AgroNT promoter and terminator source tables with CNN baselines, source-backed family links and separate assay comparisons.",
    "Link verified model identities and expose downstream evaluations separately; add 1,620 source-checked numerical rows from 13 bounded primary-source comparisons. Preserve quoted evidence and quarantine conflicting mRNABench localization cells.",
    "Count reviewed metrics from the same source evaluation setup together, retaining all original record IDs and links.",
    "Load individual source-scoped charts, remove cross-protocol pooled rankings, and move findings before detailed instructions.",
    "Review 69 selected profile facts across 18 profiles against pinned primary sources, including exact configuration, split, coverage and uncertainty distinctions. Preserve every result value and keep unresolved metadata explicit.",
    "Add two bounded ProteinGym protocol profiles with 24 sourced facts, distinguishing the 217-assay track from one complete AMFR assay. Preserve numerical results, source-authentication gaps and independent-reproduction limits.",
    "Name nine evaluated methods printed only as an author-year citation (eight ATOM3D comparisons) or an author surname (HEST, Ciga), from the benchmark text and the cited original sources. The ATOM3D RSR scorer is named only as a Rosetta scoring function; its citation conflict and settings stay unresolved. Printed labels remain searchable and auditable; IDs, values, locators and comparison conditions are unchanged. Add sourced DeepDTA and DeepAffinity method profiles.",
    "Preserve archived release bytes, historical URLs and MFASS history. Contribution intake remains controlled separately from catalogue publication.",
    "Add verified research manifests, frozen readiness assessments and reviewed-only investigation contracts. Record IDs and values remain unchanged.",
    "Add a second, additive BRCA1/BRCA2 germline-interpretation evidence intake (issue #343): 42 new source-checked clinical-lane records from five sources (Benet-Pages 2025, So 2024, HECTOR preprint, Hu 2026, Karalidou 2022/MARGINAL source-only), distinguishing reclassification rate, tool agreement, preprint classification concordance and functional-data classification yield from clinical accuracy. Add five new scoped proxy use-case mappings against these records: two for Benet-Pages criteria-version reclassification on a fixed VUS cohort (ACMG/AMP+SVI, then ENIGMA VCEP), one for So 2024 VarSome-vs-CanVIG-UK tool agreement on conflicting BRCA1 missense calls, one for HECTOR preprint agreement with ClinGen eRepo expert calls, and one for Hu 2026 BRCA2 exon 15-26 functional-data classification yield; Karalidou 2022 remains source-only with no mapping added. None of these endpoints establishes clinical accuracy, qualified human scientific review or Rewire execution. All prior scientific records, mapping objects and historic release bytes remain unchanged.",
    "Add a bounded, additive cell-type-annotation-transfer evidence intake (issue #349): Abdelaal et al. 2019 Figure S10 Panel B (34-population inter-dataset, cross-dataset/cross-species-where-applicable brain annotation-transfer comparison), SVMrejection row only (9 percent-unlabeled values across all nine VISp/ALM/MTG train-test combinations; VISp and ALM share a source study, GSE115746, so only the 7 of 9 combinations involving MTG are genuinely cross-study and cross-species), read directly from a page-image render of a separately-sourced supplement-PDF artifact and independently cross-checked before ingestion. Reuses the existing SVMrejection method/configuration records unmodified; adds one new source record, three new dataset records, one new protocol record, nine new evaluation records and nine new result records, plus one new scoped proxy use-case mapping. Scored denominator is explicitly unreported; Table 2's raw per-dataset cell counts are recorded as context only, not a confirmed denominator. Not a known-type accuracy claim, not an unknown-type-detection accuracy claim, and not a superiority claim across methods. All prior scientific records, mapping objects and historic release bytes remain unchanged.",
    "Add a bounded, additive EGFR-NSCLC evidence-retrieval proxy intake (B345/BL345, rewire-benchmarks #25 and #28; use-case-egfr-nsclc-actionability-resistance-evidence): CIViC-Fact v3's post-cutoff within-linked-publication passage-retrieval evaluation (150 candidate CIViC entries submitted/revised 2026-03-03 to 2026-06-09, reduced by sequential source-defined exclusions to 40 scored entries). Nine new clinical-lane records (one source, one dataset, one protocol, two configurations, two evaluations, two results) and one new scoped proxy use-case mapping, alongside the unchanged existing CIViC MCP proxy mapping for the same use case. Both retriever configurations (a fine-tuned MedCPT cross-encoder and a pretrained Qwen3-Reranker-8B) report two linked-paper passage-retrieval results of 37/40 (92.5%) appropriate-content rate on the identical cohort, not two independent cohorts. This is retrieval within each entry's already-identified source publication, not open-corpus literature search, and establishes no EGFR- or NSCLC-specific validation, no clinical efficacy claim, and no close of the benchmark-execution gap tracked separately at rewire-benchmarks #28. Reviewer count, blinding and replicate/seed structure for the manual retrieval judgment are unreported. The code/data repository's root licence metadata is null with no root LICENSE file, but a nested data_builder/LICENSE (GPLv3) exists at a pinned commit; its scope over the dataset/model/experiment components is unestablished and not assumed, and no code/data/model copies were ingested. All prior scientific records, mapping objects, use cases and historic release bytes remain unchanged.",
    "Add a bounded, additive genetic-perturbation-response evidence intake (B334/BL334, rewire-benchmarks #25; focused execution gap rewire-benchmarks #29; use-case-genetic-perturbation-response): PertEval-scFM (Wenteler et al., ICML 2025) Table 1, Norman single-gene section, 2,000 HVGs, three printed AUSPC endpoints (GEARS trained from scratch 0.815+/-0.039, an Xc-plus-co-expression MLP baseline 4.484+/-0.299, a context-mean baseline 4.612+/-0.317, all x10^-2), scoring the perturbation-effect delta=P-Xc rather than raw post-perturbation expression. Twelve new experimental-lane records (one source, one dataset, one protocol, three configurations, three evaluations, three results) and one new scoped proxy use-case mapping, alongside the two unchanged existing GEARS Supplementary Table 6 mappings for the same use case. This is a second, independent, from-scratch GEARS evaluation on a differently-preprocessed dataset, split mechanism and metric from Table 6; the two are not merged. Establishes no prospective experiment-selection hit rate and does not close the focused execution gap tracked separately at rewire-benchmarks #29. All prior scientific records, mapping objects, use cases and historic release bytes remain unchanged.",
    "Add a bounded, additive AMP evidence intake (rewire.it#365; rewire-benchmark-data issues #10-19): nine new use-case definitions (tumour-dna-somatic-variant-detection, tumour-rna-fusion-detection, patient-rna-splicing-validation, diagnostic-dna-pathogen-identification, diagnostic-rna-pathogen-detection, plasma-ctdna-fragmentomics, plasma-ctdna-methylation, cnv-detection-characterisation, diagnostic-genomics-model-execution), transcribed from three research groups' Codex-checked primary-source candidate dossiers (Lancet/Strelka2 virtual-tumour SNV/indel calling; EnFusion Seraseq synthetic reference standard plus a separate NCH clinical-ascertainment mapping; FRASER known pathogenic splicing-event recovery; Karius plasma microbial cfDNA; UCSF respiratory RNA mNGS; DELFI ctDNA fragmentomics; cfMethyl-Seq ctDNA methylation; DRAGEN 4.2 CNV and runtime baselines), plus a bounded Feng et al. 2025 DNA-foundation-model table expansion (Table 1 Acceptor and Donor as two separate tasks, Table 2 Arabidopsis TATA and NonTATA as two separate tasks, Table 5 pathogenic-versus-common SNP AUC/Cohen's d, Table 6 eQTL/sQTL/paQTL/ipaQTL as four separate tasks, 79 task-scoped evaluations and 138 results total) reusing the existing evidence-expansion-dna-foundation-models-2025-5d8ca9bc source unchanged, mapped as nine proxy mappings onto existing use cases (two each for splicing-follow-up and plant-promoter-reporters, one for rare-disease-candidate-ranking, four for regulatory-variant-gene-follow-up). Every preserved source conflict (Lancet/Strelka2 truth-indel counts, EnFusion Table 2 v2/v3 caption and 43.6%/43.8% precision conflicts, DELFI classifier-input wording, UCSF composite-PPA conflict, DRAGEN CNV comparator conflict) stays explicit and unresolved by inference. Automated source review, independently Codex-checked across two review cycles; no qualified human scientific review, no independent experimental reproduction, no broad clinical validation claimed. The 17 prior use cases and 72 prior mapping objects remain byte-identical."
  ],
  "research_lanes": 9,
  "search_entries": 511,
  "legacy_papers": 100,
  "legacy_result_rows": 149,
  "source_inputs": [
    {
      "file": "data/entities/baselines.jsonl",
      "sha256": "1dd9dbb7cb3169bab096e890fdb67e9266dadc97ac1961fbd369ffb19e2f8b17"
    },
    {
      "file": "data/entities/benchmarks.jsonl",
      "sha256": "10045db23430223473a6ba839871de03eabec802c092009d894f61aa78153503"
    },
    {
      "file": "data/entities/configurations.jsonl",
      "sha256": "5818979cefbe13c50d970419754c68443cd5bed7a618d623968faf8ef77cc446"
    },
    {
      "file": "data/entities/dataset-subsets.jsonl",
      "sha256": "516338dd5a8b3829765fdad381d759c436d8ace30fe87af72508896c8cb41c8e"
    },
    {
      "file": "data/entities/datasets.jsonl",
      "sha256": "f98b875ec9ada0ac41c7c2672d1c56a5422dac1faacd455105abc40bf735f106"
    },
    {
      "file": "data/entities/evaluators.jsonl",
      "sha256": "ce75e8e4b4c153b0e8d85e861cbc8da2f50ef92325a9829829de44c445d9b2bd"
    },
    {
      "file": "data/entities/methods.jsonl",
      "sha256": "ceef477ea45ec9b3e5c857feddd92495d3f7f84211bd439397ad210b2be201c4"
    },
    {
      "file": "data/entities/models.jsonl",
      "sha256": "5e21266f18a5be61e924b50dfc9fbad97ad16e303159383e020559cf8b0f5cad"
    },
    {
      "file": "data/entities/pipelines.jsonl",
      "sha256": "c2f846fb6c91bee48415474c3b0ed623875a94f8a04c74f9f80b02772fb4ddd1"
    },
    {
      "file": "data/entities/protocols.jsonl",
      "sha256": "5b2b78687beef74f24215d89f51c0fb8bf566ca4dbef12e0c1e0094722fc3b1a"
    },
    {
      "file": "data/entities/services.jsonl",
      "sha256": "bf366116db97087ba9855ee51bb2892f18404f5df2a84522e5ff9e26cd5f8024"
    },
    {
      "file": "data/entities/sources.jsonl",
      "sha256": "939db72b817609a4d386fd896a7202c353680d222dacb495b93fe87e514ac1fc"
    },
    {
      "file": "data/entities/tasks.jsonl",
      "sha256": "07ed972269aee155ce9bfde3ccd2e7a65032b90831a0f70e5e29bbc1fba09deb"
    },
    {
      "file": "data/entities/use-cases.jsonl",
      "sha256": "1cccc75b1ba319b49bcce0a6fe79be6492e09a48794f20de433da472b7fcf0b8"
    },
    {
      "file": "data/evidence/claims.jsonl",
      "sha256": "84cff02e3458bdf1aaa2aa169f22209dc26cd370847608ca34285c326ae20ead"
    },
    {
      "file": "data/evidence/evaluations.jsonl",
      "sha256": "4642e4f57517a14eeda5d448171f77ea019c194953cc566aeeb6e54fdda1b52a"
    },
    {
      "file": "data/evidence/results.jsonl",
      "sha256": "e9f364904b7de8bd17a747823a4d430297d5200227391c2b34ef860d99a9fa9a"
    },
    {
      "file": "data/provenance/records.jsonl",
      "sha256": "96890532729ca279513e5306b9148be53ffa5e3cb31f8888cf8de0feb013aae1"
    },
    {
      "file": "data/omics/use-cases/sources/mfass-matched-study-intake.md",
      "sha256": "0af394e74888c01dbe2970d2e28b9bbacb4a83d355881feaaa3a704396a2074d"
    },
    {
      "file": "data/omics/use-cases/sources/amfr-pilot-readme.md",
      "sha256": "a3208aa4537e8d28c56f68f4559299f31241c50b8d8b2b3776cba5f1718caa43"
    },
    {
      "file": "data/omics/use-cases/sources/clinical-priorities-2026-09-28.md",
      "sha256": "57ebf436d7d92d349700d7f3cb533b80b88e4e15d0d3129e824bfa3b06ecdab7"
    },
    {
      "file": "data/omics/use-cases/sources/research-priorities-2026-09-28.md",
      "sha256": "1f7c3d2d1e0ba852e4ae725bec2e77e55e0aa81496551242e3fbe9fee7662ae0"
    },
    {
      "file": "data/omics/use-cases/review.json",
      "sha256": "c97804a941ad497d96d18f24ef944286021cd8603678a1088c45544c834c908e"
    },
    {
      "file": "data/research/manifests.json",
      "sha256": "bb5b7f5c8c26d1e1318706243135db8f6e9e5afbda48bf87941535f06c4b8662"
    },
    {
      "file": "data/research/investigations.json",
      "sha256": "37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570"
    },
    {
      "file": "data/omics/search-ledger.jsonl",
      "sha256": "043dd9746be19d4e8489a662eeae69cdd10255ae4c1d2ef33a8967359c9ff26f"
    }
  ],
  "use_cases": {
    "schema_version": "1.0",
    "input_sha256": "35f52add6b109da224409bbf706eb584e4b963c80a0c952cc0448d88fc9b3bfc",
    "use_cases": 26,
    "mappings": 376
  },
  "use_case_sources": [
    {
      "file": "use-case-source-0af394e74888c01dbe2970d2e28b9bbacb4a83d355881feaaa3a704396a2074d.md",
      "sha256": "0af394e74888c01dbe2970d2e28b9bbacb4a83d355881feaaa3a704396a2074d"
    },
    {
      "file": "use-case-source-1f7c3d2d1e0ba852e4ae725bec2e77e55e0aa81496551242e3fbe9fee7662ae0.md",
      "sha256": "1f7c3d2d1e0ba852e4ae725bec2e77e55e0aa81496551242e3fbe9fee7662ae0"
    },
    {
      "file": "use-case-source-57ebf436d7d92d349700d7f3cb533b80b88e4e15d0d3129e824bfa3b06ecdab7.md",
      "sha256": "57ebf436d7d92d349700d7f3cb533b80b88e4e15d0d3129e824bfa3b06ecdab7"
    },
    {
      "file": "use-case-source-a3208aa4537e8d28c56f68f4559299f31241c50b8d8b2b3776cba5f1718caa43.md",
      "sha256": "a3208aa4537e8d28c56f68f4559299f31241c50b8d8b2b3776cba5f1718caa43"
    }
  ],
  "research_schema_version": "1.0",
  "total_records": 67714,
  "public_records": 67462,
  "counts": {
    "model": 247,
    "method": 448,
    "configuration": 4301,
    "pipeline": 125,
    "service": 6,
    "benchmark": 82,
    "task": 495,
    "protocol": 1018,
    "evaluator": 4,
    "dataset": 550,
    "dataset_subset": 409,
    "baseline": 83,
    "evaluation": 17369,
    "result": 38613,
    "source": 1257,
    "claim": 2429,
    "use_case": 26
  },
  "quarantined_results": 222,
  "scope_excluded_records": 27,
  "source_checked_results": 38597,
  "rewire_result_rows": 12,
  "scope": "Specialist omics and molecular models; no clinical assistants or standalone medical imaging.",
  "limitation": "A dated discovery and source-transcription review, not an exhaustive census or independent reproduction of external experiments."
}