onc_clintrial_RAG

🧬 Clinical Trial Retrieval-Augmented Generation (RAG) 🔬

Retrieval-Augmented Generation (RAG) system for oncology clinical trial question answering using Gemma-3-4B-IT, FAISS, and Qwen3-Embedding-0.6B.

---

Table of Contents

• Introduction
• Data
• Methodology
• Evaluation
• Usage and Intended Use
• Prompt Format
• Expected Output Format
• Limitations

---

INTRODUCTION

Clinical trial data is updated in a frequent and irregular manner, making it challenging for clinicians and researchers to stay current on developing therapeutics. Trial records hosted on ClinicalTrials.gov are updated in a continuous and ad hoc manner, following no fixed schedule. While public, this data is not easily accessible by LLMs or search engines, and in combination with the complex domain terminology, makes it easy to misrepresent this information. Furthermore, commonly used LLMs rely primarily on trained information, leading to stale information being included in responses and an inability to answer with accurate information.

This leads to increased work for physicians and researchers who must manually read through trial records to be aware of scientific advancements and inform the best clinical decisions. An automated tool designed to reduce the manual labor to monitor clinical trials would save time for busy clinical teams and allow for more accessible trial enrollment. This efficiency matters in fast-paced areas of research like oncology, where new treatments are showing up frequently, and patients are interested in being enrolled in clinical trials.

Here, I take a strong base model (Gemma-3-4B-IT) and build out the retrieval side of the RAG using FAISS and Qwen3 embeddings to allow the model to access updated trial information, without the need for retraining. RAG improved accuracy on clinical trial test set from 0.17 without retrieval to 0.89 with retrieval and outperformed two similar size comparison models on this task (0.50 and 0.26). This code uses a subset of oncology clinical trials to show how in practice, an advanced system could be built out for instant and widespread clinical trial information retrieval.

Key Result: Retrieval improved clinical trial question answering from 0.168 to 0.888, achieving Recall@3 = 0.975 on the held-out clinical trial evaluation set without retraining the base language model.

---

DATA

To build a document store, I downloaded 600 oncology clinical trials using keyword search via API from ClinicalTrials.gov. After deduplicating and reformatting the data into a Pandas data frame retaining the most important fields, 410 unique trials remained. Gemini API was used to generate 161 question/answer pairs, tied to 131 unique trials, meaning some trials have more than one associated question. Since this is a RAG task, there is no direct test/train split; all 410 trials were used for the document retrieval store, and the 161 question test set was held out for final RAG evaluation. The 410 trials are in the clintrials_documents.csv file and the 161 question/trial/answer sets are in clintrials_test.csv file. During model building, a discrepancy was identified that initially the RAG did not have access to the eligibility information in the trial, which has been fixed by directly re-embedding the eligibility data. The Gemini API prompt and a sample test data point is shown below.

Gemini Prompt

prompt = f"""
    You are an oncology expert. Analyze these 200 clinical trials and generate exactly 20 realistic clinical questions.

    Each question must be something a medical doctor or scientific researcher would actually ask (e.g., matching a patient based on a cancer type, trial phase, or specific eligibility rule).

    Spread the questions out evenly across different trials in this specific list.

    For each item in your JSON output, provide:
    1. "target_id": The exact NCT ID or Trial Index.
    2. "question": The complex clinical query.
    3. "ground_truth_summary": An ideal 2-4 sentence summary answering the question based strictly on this trial.

    CRITICAL SUMMARY REQUIREMENTS:
    - The summary must include the drug or intervention, the disease or condition being studied, the clinical trial phase, the recruitment/trial status, and the primary objective of the study.
    - Mention other important details if available and relevant, such as the study design, sponsor, or key eligibility criteria.
    - Write in clear, professional language and strictly do not exceed 4 sentences.

    Here are the trials:
    {trials_text}
"""

Example Evaluation Sample

---

METHODOLOGY

Model Architecture

This RAG architecture was built using the baseline model Gemma-3-4B-IT. I previously tested three embedding and distance metric combinations of Qwen3-Embedding-0.6B and MiniLM-L6-v2 embeddings with cosine and Euclidean distance metrics, and ultimately selected Qwen3-Embedding-0.6B with cosine similarity based on retrieval accuracy. Prior to implementing RAG, the model's pre-document retrieval performance was assessed to gather a baseline. Then, documents were split into 800-token chunks with 10% chunk overlap to reduce the likelihood of the model losing relevant information split across document chunks. FAISS-based vector retrieval was implemented, and model performance was measured with the top-k retrieved documents added to the prompt. I tested both k=3 and k=5 as retrieval depths with the initial thought that k=5 would improve retrieval accuracy, however, it had no beneficial effects on datasets other than HotPotQA thus I chose k=3 for my final implementation across all datasets.

Final configuration

• Generator: Gemma-3-4B-IT
• Embedding model: Qwen3-Embedding-0.6B
• Similarity metric: Cosine similarity
• Vector database: FAISS
• Chunk size: 800 tokens
• Chunk overlap: 10%
• Retrieval depth: k = 3

Experiments

Both document retrieval and answer accuracy were assessed, with answer accuracy being measured by an F1-similar metric that compares the model generated responses to the ‘gold’ answer. Additionally, I tracked refusal rates, where the model declines to answer or cites missing documents, to compare where the model did not retrieve the true document (as shown by recall) to when the model believed it did not retrieve the true document. Two comparison models were evaluated, Qwen2.5-3B-Instruct and Phi-3.5-mini-instruct, on their ability to answer the questions in the absence of document retrieval, to test whether the RAG implementation improved performance relative to other strong base models of similar size. Lastly, I tested my RAG architectures across three benchmarks to assess its ability to generalize to other question/document/answer sets beyond the clinical trial task. Note, the full implementation and evaluation pipeline is available as train_eval_pipeline.py in this repository.

To summarize, I conducted four experiments:

1. Does RAG improve the performance of the baseline model to answer clinical trial specific questions?
2. Does the RAG model answer clinical trials questions in comparison to other strong base models?
3. Does the RAG model generalize well to other document/answer sets?
4. How does the RAG model recognize the limits of its retrieval? (Recall vs Refusal)

---

EVALUATION

To evaluate model performance, the base model, RAG model, and comparison models were tested using the clinical trial test data and three Hugging Face benchmark datasets: Natural Questions, HotPotQA, and BioASQ. Natural Questions consists of real user queries to Google and Wikipedia pages used to answer those questions, intended to test general knowledge and information retrieval. HotPotQA is a benchmark requiring models to synthesize data across multiple documents by using multi-hop reasoning. Lastly, I chose a benchmark focused on retrieval and answer summarization in the biomedical domain using BioASQ to evaluate technical evidence-based responses. This dataset pairs questions with relevant clinical passages aimed to emulate PubMed abstracts. Performance on test data was evaluated against two other base models, Qwen2.5-3B-Instruct and Phi-3.5-mini-instruct, based on being openly available instruction tune models of a similar size.

The metrics used to evaluate model performance were Recall@3 for document retrieval and an F1-style metric that computes precision and recall based on overlap between generated response and correct answer, filtering out un-important ‘stop’ words. This F1 approach was chosen to balance penalizing overly verbose answers with providing enough relevant information. RAG implementation of Gemma-3-4B-IT shows significant improvement in accuracy over the base model performance. Notably, the model had exceptional Recall@3 (0.975) and minimal refusals (2 out of 161 test questions). This RAG architecture had moderate generalizability. It generalized well to Natural Questions, which had 0.54 answer accuracy, high recall of 0.956, and minimal refusals (84 out of 2474 test questions). The model failed to generalize well to HotPotQA and BioASQ, likely due to the multi-hop reasoning and biomedical domain specificity required. Furthermore, RAG significantly outperformed the other two pre-retrieval models on the clinical trial task. Across datasets, the comparison models did perform interestingly well on benchmark tasks without document retrieval, such as Qwen2.5-3B-Instruct scored 0.357 on BioASQ.

Accuracy by Dataset and Model

Retrieval Recall@3 (RAG Model Only)

Refusal Rate (RAG Model)

---

INTENDED USE AND USAGE

The intended use for this code and this architecture is for researchers or physicians who want quick answers about current clinical trials. It is not meant to be used as medical advice, and all information should be used under the guidance of a clinical professional.

The code below loads the saved model + FAISS index and answers new questions in the same manner.

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
from langchain_community.vectorstores import FAISS
from langchain_huggingface import HuggingFaceEmbeddings

HF_id = "clairesull/onc_clintrial_RAG"

# Load the generator/base model
tokenizer = AutoTokenizer.from_pretrained(HF_id, subfolder="generator")
tokenizer.padding_side = "left"
tokenizer.pad_token_id = tokenizer.pad_token_id or tokenizer.eos_token_id
model = AutoModelForCausalLM.from_pretrained(HF_id, subfolder="generator", device_map="auto", torch_dtype=torch.bfloat16)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=250, do_sample=False, return_full_text=False)

# Load the FAISS retrieval index for clinical trials
# Alternatively, can load the FAISS index for benchmarks: faiss_nq, faiss_hotpotqa, faiss_bioasq
hf_hub_download(repo_id=HF_id, filename="faiss_clinical_trials/index.faiss")
faiss_dir = hf_hub_download(repo_id=HF_id, filename="faiss_clinical_trials/index.pkl").rsplit("/", 1)[0]
embedding_model = HuggingFaceEmbeddings(model_name="Qwen/Qwen3-Embedding-0.6B")
vector_db = FAISS.load_local(faiss_dir, embedding_model, allow_dangerous_deserialization=True)

# Retrieve relevant trial documents and generate an answer
question = "What are the eligibility criteria for a Phase 0 study involving Temsirolimus?"
docs = vector_db.similarity_search(question, k=3)
context = "\nExtracted documents:\n" + "".join(f"Document {i} (Source: {d.metadata.get('source', 'unknown')}):::\n{d.page_content}\n" for i, d in enumerate(docs))

clinical_specific_instruction = (
    "The documents above may describe different clinical trials. First, internally "
    "identify which document (by Trial ID) is most relevant to the question. Then, "
    "without restating which document you chose, answer the following question using "
    "only that document with a 2-4 sentence summary. The summary must include the drug "
    "or intervention, the disease or condition being studied, the clinical trial phase, "
    "the recruitment/trial status, and the primary objective of the study."
)

raw_prompt = f"Context:\n{context}\n\n{clinical_specific_instruction}\nQuestion: {question}\nAnswer:"
prompt = tokenizer.apply_chat_template([{"role": "user", "content": raw_prompt}], tokenize=False, add_generation_prompt=True)

output = pipe(prompt)[0]["generated_text"].strip()
print(output)

---

PROMPT FORMAT

For clinical trial data, inform the model to retrieve the most relevant document, then answer the question using only that document. Tell the model to format a 2–4 sentence answer and include relevant pieces of information including the drug, the patient condition, phase of the trial, primary outcomes, etc. For all other benchmarking datasets (Natural Questions HotPotQA and BioASQ), provide simple instructions to answer the question directly using the provided context.

clinical_specific_instruction = (
    "The documents above may describe different clinical trials. First, internally "
    "identify which document (by Trial ID) is most relevant to the question. Then, "
    "without restating which document you chose, answer the following question using "
    "only that document with a 2-4 sentence summary. The summary must include the drug "
    "or intervention, the disease or condition being studied, the clinical trial phase, "
    "the recruitment/trial status, and the primary objective of the study. Where "
    "relevant, mention the study design, sponsor, or key eligibility criteria."
)

general_instruction = (
    "Answer the following question directly and concisely, strictly using the context above."
)

---

OUTPUT FORMAT

The expected output is a 2–4 sentence response that summarizes the relevant clinical trial and answers the question. The format should be a plain text summary. The response should mention things such as the therapeutic or intervention, the patient disease, the study phase or status.

Example Question

What are the age and health requirements for pediatric patients with Acute Lymphoblastic Leukemia to participate in a quality of life study during maintenance chemotherapy?

Example Response

Document 0 (NCT03132948) outlines the age and health requirements for pediatric patients with Acute Lymphoblastic Leukemia (ALL) participating in a quality of life study during maintenance chemotherapy. The study includes children aged 8-18 years who are able to speak and read English and have no contraindications to moderate physical exercise, as determined by their pediatric oncologist. Furthermore, participants must have parental consent and patient assent, and not have any documented psychiatric or neurological disorders that would interfere with study participation.

---

LIMITATIONS

• Benchmark HotPotQA assesses multi-hop reasoning and requires connecting information from two separate documents, retrieval of 0.65 indicates the retrieval often missed one of the two documents needed to answer the question. A higher retrieval number of k=5 was shown to slightly improve performance, but not significantly. Failure to retrieve enough relevant documents limits the ability of the RAG to answer the question accurately.

• BioASQ recall was very low (0.28), which inevitably impacted accuracy, however, accuracy is slightly retained (0.37), like the accuracy of the Qwen2.5-3B model (0.35) which had no document retrieval. This suggests the base models can answer these questions correctly without retrieval knowledge, reducing the impact of this benchmark in RAG evaluation.

• Experiments were run to test different answer-level accuracy metrics: count of number of correct words over estimated accuracy, a stricter metric (BERTScore semantic matching) underestimated accuracy, the final F1 metrics was best option, but could still be improved by a LLM judge.

• Mismatch in eligibility data, the Gemini API had access to the full eligibility for each trial to generate the ‘gold’ response whereas the RAG had only the trial summary, not necessarily including the eligibility. This was remedied by adding eligibility information into the embedded text, but the gold answer and model generated answer were not produced based on identical prompts.