rewirebio.iobenchmarks

Use caseDNA and genomesResearch and clinical research

Review EGFR lung-cancer actionability evidence

Which systems retrieve correctly scoped sensitivity and resistance evidence for advanced EGFR-mutant NSCLC?

What the evidence shows

Two sources add multi-system comparisons on precision-oncology evidence tasks.

Neither is specific to EGFR-mutant NSCLC after targeted therapy, and neither tests treatment choice or patient benefit. Lin et al. 2025 (npj Precision Oncology), whose authors did not develop the models, asked GPT-4o, Llama 3.1 and Qwen 2.5 to assign knowledge-base evidence levels to queries naming a gene, an alteration and a tumour type but no drug. Mean top-1 accuracy over 100 runs in assigning OncoKB levels, which include the resistance levels R1 and R2, was 0.3393, 0.3066 and 0.3328; for CIViC levels it was 0.1865, 0.1212 and 0.2485.

The authors note that the public OncoKB and CIViC tables may be in the models' training data. In the TREC 2020 Precision Medicine track, scored by the track organisers, systems ranked MEDLINE abstracts for 40 cancer, gene and treatment topics, two of them EGFR-mutant NSCLC with afatinib or osimertinib. For topical relevance the best printed runs reached infNDCG 0.5325 (a BM25 baseline tuned on 2019 data) and 0.5303 (BioBERT re-ranking).

For ranking by evidence tier, the best printed automatic run reached NDCG@30 0.4238 (monoT5 with trial and meta-analysis terms added to the query); two manual runs that used human relevance input are not counted. The overview prints only the best run of the top five teams per metric, per-topic EGFR scores are not tabulated, and the topics carry no variant, prior-therapy or resistance context. No source measures retrieval of source-linked resistance evidence after progression on an EGFR inhibitor.

You bring
  • Interpreted biomarker, genome build/transcript, histology, stage and co-alterations
  • Prior therapies, progression and sampling dates, including tissue or plasma source
  • Jurisdiction, evidence cutoff and a versioned source corpus
You want
Source-linked evidence assertions with regimen, sensitivity or resistance direction, treatment setting, native evidence tier, contradictions and unresolved information.
Evidence in this release
4 comparisons shown · 15 tools · 23 configurations
Kept up to date
Evidence last checked 9 October 2026 · re-checked monthly, next due 9 November 2026
Executive Takeaway

For "Review EGFR lung-cancer actionability evidence", on LLMs assigning knowledge-base evidence levels (Lin et al. 2025), OncoKB levels 1-4, R1, R2 (Top-1 accuracy, a proxy protocol), GPT-4o scored 0.3393, reported by Benchmarking large language models GPT-4o, llama 3.1, and qwen 2.5 for cancer genetic variant classification.

Best reported valueGPT-4oBenchmarking large language models GPT-4o, llama 3.1, and…
Top-1 accuracy0.3393LLMs assigning knowledge-base evidence levels (Lin et al. 2025), OncoKB levels 1-4, R1, R2
Baseline on this protocolNone scoredNo margin is computed
Ranked tools66 from independent studies

6 tools have a value on this protocol; 6 of them come from studies whose authors did not develop the tool. No baseline method is scored on this protocol. Reviewers judged this protocol a proxy for the use case.

Ranking

Ranked on LLMs assigning knowledge-base evidence levels (Lin et al. 2025), OncoKB levels 1-4, R1, R2 (Top-1 accuracy, higher is better). Proxy protocol Reviewers judged this protocol a proxy for the use case, not a direct test of it. Only results on this protocol are ranked; other tools are listed below the table.

3 tools in 6 rows, from 1 source. 6 of 6 results (100%) come from an independent study.

Rank▲ModelTop-1 accuracy (higher is better)EvidenceCoverageReported byLicense
#10.3393Source-checked Independent study1/4Benchmarking large language models GPT-4o…Not recorded
#2
Qwen2.5Qwen 2.5 72B, basic prompt, temperature 0.8 (default)
0.3328Source-checked Independent study1/4Benchmarking large language models GPT-4o…Not recorded
#3
Llama 3.170B, basic prompt, temperature 0
0.3312Source-checked Independent study1/4Benchmarking large language models GPT-4o…Not recorded
#4
Llama 3.170B, basic prompt, temperature 0.4
0.3178Source-checked Independent study1/4Benchmarking large language models GPT-4o…Not recorded
#5
Llama 3.170B, basic prompt, temperature 0.8 (default)
0.3066Source-checked Independent study1/4Benchmarking large language models GPT-4o…Not recorded
#6
Qwen2.5Qwen 2.5 72B, refined prompt, temperature 0.8 (default)
0.2994Source-checked Independent study1/4Benchmarking large language models GPT-4o…Not recorded
12 tools are evaluated elsewhere in this use case but not on this protocol, so not ranked

Baseline

BM25 keyword retrieval. BM25 keyword ranking is the non-learned retrieval method that the neural rerankers in TREC 2020 Precision Medicine are compared with. The method record is team ASCFDA, whose only run used Anserini BM25 with default parameters; BM25 itself, and the collection plan's knowledgebase lookup with manual review, have no method record.

Machine-learning methods are compared with the baseline on the same data in one table: Precision-oncology literature retrieval systems (TREC 2020 Precision Medicine). Baseline rows are marked in each table.

Tools compared

Comparisons

Unified model comparison for this use case. Compare tools against protocol baselines in a single view.

LLMs assigning knowledge-base evidence levels (Lin et al. 2025)

Proxy evidenceIndependent study: Reported in a study whose authors did not develop the evaluated tool.

Top-1 accuracy (higher is better), with the other metrics below each value. The best value in each column is highlighted.

ToolOncoKB levels 1-4, R1, R2CIViC levels A-E
GPT-4ofoundation model
0.3393 (best in column)0.1865
Qwen2.52 sweepsQwen 2.5 72B, basic prompt, temperature 0.8 (default)foundation model
0.33280.2485 (best in column)
Llama 3.13 sweeps70B, basic prompt, temperature 0foundation model
0.3312Not reported
Caveats and method

What each column measures:

  • OncoKB levels 1-4, R1, R2: Top-1 accuracy of GPT-4o, Llama 3.1 and Qwen 2.5 (and temperature and prompt variants) in assigning OncoKB levels, including resistance levels R1 and R2, to 625 OncoKB associations
  • CIViC levels A-E: Top-1 accuracy of GPT-4o, Llama 3.1 and Qwen 2.5 (and a refined Qwen prompt) in assigning CIViC evidence levels to 4,426 CIViC associations
  • Pan-cancer variant set; no EGFR- or NSCLC-specific score is reported.
  • The model assigns an evidence level from gene, alteration and tumour type alone; no prior therapy, line or evidence date is given and no source is cited.
  • Public OncoKB and CIViC tables may be in the models' training data (Discussion P33).
  • Queries name no drug, so a variant with both a sensitivity and a resistance level for different drugs has no single correct answer; the paper does not say how such queries were scored.
  • Evidence levels are assigned from the model's own knowledge; except in one FoundationOne run, nothing is retrieved and no source is returned.
  • Queries name no drug, so a variant with both several evidence items at different levels has no single correct answer; the paper does not say how such queries were scored.

Protocols: Lin et al. 2025 OncoKB level-of-evidence assignment, Lin et al. 2025 CIViC evidence-level assignment

Primary benchmark evaluation shown (LLMs assigning knowledge-base evidence levels (Lin et al. 2025)). 3 additional study protocols (CIViC evidence-direction retrieval, March 2026, CIViC-Fact v3 within-linked-publication passage retrieval, post-cutoff cohort, Precision-oncology literature retrieval systems (TREC 2020 Precision Medicine)) are cited in individual tool drawer rows and under Question, sources and review.

Not covered yet

  • No independently adjudicated advanced EGFR-mutant NSCLC evidence-retrieval benchmark with prior therapy, treatment line, date, jurisdiction and contradictions was established by this bounded search.
  • CIViC study reports pan-cancer aggregate and predictive direction metrics; no published EGFR-only score was found.
  • OncoTraj (arXiv:2606.11144) predicts longitudinal resistance and is outside the requested evidence-retrieval endpoint.
  • Manual versioned lookup plus primary-source review still needs measured equal effort.
  • Qualified human scientific review remains outstanding; automated source transcription does not establish experimental replication.
  • Multi-system comparisons found are pan-cancer: LLM evidence-level assignment (Lin et al. 2025) and literature ranking for cancer, gene and treatment topics (TREC 2020 Precision Medicine, two EGFR topics); neither prints an EGFR-only score.
  • TREC 2020 Precision Medicine scores for the two EGFR topics appear only as figures in the run appendices, and the topics name a gene and a drug (afatinib, osimertinib) but no variant, prior therapy or resistance question.
  • LLM evidence-level accuracy may be inflated by training-data exposure to the public OncoKB and CIViC tables (Lin et al. 2025 Discussion), and its queries name no drug, so sensitivity and resistance levels for the same variant are not separated.
  • TREC 2020 Precision Medicine Phase 2 (evidence-tier ranking) values rest on the overview alone; no public per-run NDCG@30 was found to check them.
  • No multi-system comparison of retrieving resistance evidence (for example T790M, C797S or MET amplification) after progression on an EGFR inhibitor was found.

Contribute evidence or propose a correction

How to cite

Review EGFR lung-cancer actionability evidence. Rewire biological model benchmark database, release 2026-11-21-c8bea25e5abb. https://benchmarks.rewirebio.io/use-cases/egfr-nsclc-actionability-resistance-evidence/

Release 2026-11-21-c8bea25e5abb. Rankings and values change between releases, so keep the release id in the citation. Cite the original publication as well for any value you reuse: each result links its source.

The compilation and annotations made by Rewire in this release are offered under CC BY 4.0; values and text taken from cited sources stay under the terms of those sources and are not covered by this licence. Data licence

Question, sources and review

Who this is for, setting and exclusions

Choose an evidence-review system that connects tumour findings to relevant sensitivity and resistance studies while exposing contradictions and missing context.

Who this is for
  • Molecular tumour-board scientists
  • Oncologists and clinical molecular scientists reviewing evidence
Biological setting
Advanced EGFR-mutant non-small-cell lung cancer, including progression after EGFR-targeted therapy. Treatment history and evidence date are part of every comparison.
Clinical research scope
Clinical research on support for qualified molecular-oncology evidence review. Applicability depends on tumour, biomarker, prior therapy, sampling context, jurisdiction and evidence date.

Outside this use case

  • Evidence retrieval does not select a treatment or establish individual treatment benefit.
  • Trial eligibility, response and survival require separate evaluations.
  • Missing evidence does not establish resistance, and evidence tiers from different systems are not interchangeable.
  • Treatment-recommendation concordance (for example LLM agreement with NCCN guidelines or tumour-board decisions) is treatment selection, not evidence retrieval.
Evidence collection plan
Evidence collection planCollecting evidence

Already on record: 4 comparisons shown · 15 tools, in the evaluated evidence above. The status above describes only the specific comparison in this plan, which remains open; it does not mean no evidence has been collected. The plan defines a comparison to investigate; it does not establish model performance or suitability.

Comparison question

Does retrieval-assisted review recover more correctly scoped sensitivity/resistance evidence with fewer unsupported assertions than a versioned lookup/manual-search workflow at equal review effort?

Baselines, outcomes and validation requirements

Baselines to include

  • Versioned knowledgebase lookup plus conventional manual primary-source review

Outcomes to measure

  • Relevant-evidence recall and unsupported-assertion rate
  • Tumour, treatment-line, direction, date and citation fidelity
  • Contradiction handling, missing-information handling and review time

Validation requirements

  • Freeze the question, corpus and knowledge cutoff; hold out studies and molecular profiles.
  • Preserve native evidence tiers and include contradictory or inapplicable evidence.
  • Audit citation support and retain unknown clinical information.
  • Use independent qualified molecular-oncology adjudication.
  • Confirm source-specific benchmarking permissions and separate evidence review from treatment and eligibility outcomes.

Next collection task

Close the remaining endpoint, independence and comparator gaps documented in the 30 September 2026 audit and linked article issue; obtain qualified scientific review.

Sources, review and downloads

Automated source review · 2026-09-30 · Codex source curation, independent worker cross-review and root integration review

Source-backed literature curation with independent automated transcription and scope checks. No new model execution, independent experimental replication, qualified human scientific review or clinical validation.

Release 2026-11-21-c8bea25e5abb · question use-case-egfr-nsclc-actionability-resistance-evidence

Use-case input digest bfbb7ab9303becad1e3cb395568c26f41e72ddb67cd1a89a06d9eaa0f76bd95f

Download questions and review metadata (JSON) (gzip) · Verify release checksums (gzip)