llama-3.1-8b-instruct-lora-clinical_iter2_epoch2

Llama-3.1-8B-Instruct LoRA Fine-Tuned

This model is a PEFT/LoRA adapter single epoch fine-tuned on top of meta-llama/Llama-3.1-8B-Instruct. Fine tuning has been done on iter2 anchor-context-groundTruth_junkHandled dataset related to clinical trials. This fine tuned adapter on base LLM can be to address clinical trials related questions in general(macro/broad overview, patient eligiblity, operations etc related question). Further this can also be used for prompt engineering based on CT, or RAG generation(related to CT) based on retrieved chunks.

Model Details:-

Model Description:-

• Model Type: Causal Large Language Model (Autoregressive Decoder),
• Base model: meta-llama/Llama-3.1-8B-Instruct,
• Maximum Sequence Length: 131,072 tokens (128K context window),
• Output Dimensionality (Hidden Size): 4096 dimensions,
• Supported Modality: Text,
• Training Dataset: iter2 anchor-context-groundTruth_junkHandled dataset (mapped via Chat Templates),
• Language: en,
• License: llama3.1 (Subject to Meta's Llama 3.1 Community License Agreement)

Model Sources:-

• Documentation:-Transformers Documentation / PEFT Documentation,
• Repository: Transformers on GitHub,
• Hugging Face: PEFT Adapters Hub

Fine tune model archtitecture:-

PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): LlamaForCausalLM(
      (model): LlamaModel(
        (embed_tokens): Embedding(128256, 4096)
        (layers): ModuleList(
          (0-31): 32 x LlamaDecoderLayer(
            (self_attn): LlamaAttention(
              (q_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.05, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=16, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=16, out_features=4096, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (k_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.05, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=16, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=16, out_features=1024, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (v_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.05, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=16, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=16, out_features=1024, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (o_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.05, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=16, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=16, out_features=4096, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
            )
            (mlp): LlamaMLP(
              (gate_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=14336, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.05, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=16, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=16, out_features=14336, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (up_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=4096, out_features=14336, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.05, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=16, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=16, out_features=14336, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (down_proj): lora.Linear4bit(
                (base_layer): Linear4bit(in_features=14336, out_features=4096, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.05, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=14336, out_features=16, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=16, out_features=4096, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (act_fn): SiLUActivation()
            )
            (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
            (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
          )
        )
        (norm): LlamaRMSNorm((4096,), eps=1e-05)
        (rotary_emb): LlamaRotaryEmbedding()
      )
      (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
    )
  )
)

How to Load and Run Inference:-

You can load this model using bitsandbytes for 4-bit quantization and peft to apply the adapter weights. Tokenizer formatting utilizes the base model's official chat template to map query/question tokens alongside retrieved context blocks.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"

# 1. Load the tokenizer from the base model
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# 2. Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True
)

# 3. Load quantized base model
base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.float16
)

# 4. Load your current LoRA fine-tuned adapter
model = PeftModel.from_pretrained(
    base_model,
    "vab46/llama-3.1-8b-instruct-lora-clinical_iter2_epoch2"
)
model.eval()

# ==========================================
# 5. STRUCTURE RAG PROMPTS WITH CHAT TEMPLATE
# ==========================================
# Mapping the query/question with relevant context chunks into the message schema
SYSTEM_PROMPT = "You are a helpful assistant specialized in clinical data analysis."
retrieved_context = "..."  # Paste or map your retrieved context blocks here
user_query = "..."         # Paste your question/anchor here

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user", 
        "content": f"CONTEXT(RETRIEVED_BLOCKS):
CONTEXT:
TITLE: WATCH-STEP : Pilot Trial: Smartwatch-Guided Secondary Prevention After Stroke Randomized Trial of Nurse-led Program With Active vs Passive Smartwatch in Minor Stroke. A Randomized Controlled Trial Evaluating a Nurse-led Secondary Prevention and Physical Activity Program Supported by Either an Active Smartwatch (Structured Feedback) or Passive Smartwatch in Patients With Minor Stroke.
SUMMARY: After a first stroke or transient ischemic attack (TIA), the risk of recurrence is high in the weeks and months following the initial event. There are several modifiable risk factors that can reduce this risk, such as blood pressure, diet, physical activity, and smoking. Many stroke patients (NIHSS \< 5) have a low daily step count during the early recovery period, despite a good functional prognosis.
Active smartwatches provide real-time feedback, track progress, and set personalized walking goals, thereby boosting motivation and adherence to physical activity recommendations.
The combination of advice provided by nurses and active behavioral coaching supported by a smartwatch, compared to passive monitoring, could significantly increase daily step counts over a 12-week period. The results of this research will help guide future large-scale secondary prevention strategies integrating digital health and structured nursing support.
INCLUSION_CRITERIA:
Age ≤ 40-75
Patients with recent minor ischemic stroke or TIA, initial NIHSS ≤5 (\< 30 days)
mRS ≤2 before stroke and on discharge
Patient with Wifi access (for data transmission)
Able to understand smartwatch instructions
Written informed consent
Ambulatory ≥10 meters without assistance and without walking aid
Expected to live at home during the study period.

QUESTION:
What is the main goal of this clinical trial regarding secondary prevention after a minor stroke?"
    }
]

# Apply Llama 3.1 specific formatting tokens automatically
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

# Generate response
with torch.no_grad():
    outputs = model.generate(
        inputs, 
        max_new_tokens=512, 
        eos_token_id=tokenizer.eos_token_id
    )

response = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)
print("RELEVANT_ANSWER(generated by LLM):", response)

<u>Evaluation(on clinical trials test_data splitted in 0.1 ratio at seed=42)</u>:-

STRATIFIED RESULTS:

GROUNDEDNESS / COMPLETENESS DIAGNOSTICS:

Head to head base vs LoRA in Rouge-score & Bert-score:

OVERALL MACRO RESULTS:

Training Dataset:

• Samples

Training Hyperparameters

The following configurations were used during training:

• Train Batch Size: 4
• gradient_accumulation_steps: 4,
• Total Epochs: 1
• Weight Decay: 0.01
• Learning Rate: 0.0002
• Output Directory: /kaggle/working/rag3_llama_lora
• LR Scheduler Type: SchedulerType.COSINE
• Warmup Steps: 22
• FP16 Precision: True
• BF16 Precision: False
• Gradient Checkpointing: True
• Logging Strategy: IntervalStrategy.STEPS
• Logging Steps: 50
• Logging First Step: True
• Save Strategy: SaveStrategy.STEPS
• Save Steps: 100
• **#save_strategy:, "epoch",#use this (ignore previous 2 stepwise checkpoint) if strong gpu +VRAM, else use stepwise ckpt
• Save Total Limit: 1
• Evaluation Strategy: IntervalStrategy.NO
• Seed: 42
• Report To: []
• Dataloader Num Workers: 2
• Dataloader Pin Memory: True
• Remove Unused Columns: False

Training Loss History

Training loss:-0.19749066206725402