Llama-3.1-8B-Instruct-BotMed-v2

BotMed/Llama-3.1-8B-Instruct-BotMed-v2

<!-- Provide a quick summary of what the model is/does. -->

BotMed-v2 is a fine-tuned version of Llama 3.1 8B Instruct, adapted for medical chatbot use cases using LoRA.

Model Description

<!-- Provide a longer summary of what this model is. -->

BotMed-v2 is a fine-tuned version of Meta's Llama 3.1 8B Instruct, specialized for medical chatbot and question-answering use cases. It was trained using LoRA (Low-Rank Adaptation) on the ruslanmv/ai-medical-chatbot dataset, with the base model loaded in 4-bit NF4 quantization to enable efficient fine-tuning on Kaggle GPUs.

The LoRA adapters (rank 16, alpha 32) target the attention and MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), allowing the model to adapt to medical dialogue patterns while preserving the general instruction-following capabilities of the base Llama 3.1 model. Training was run for up to 3 epochs with early stopping based on evaluation loss, using a cosine learning rate schedule and the paged AdamW 8-bit optimizer.

BotMed-v2 is intended to support use cases such as retrieval-augmented generation (RAG) pipelines for medical knowledge bases, clinical communication training simulations, synthetic medical dialogue generation, and preliminary response drafting in human-in-the-loop healthcare workflows. It is not a certified diagnostic tool and should not be used as a substitute for professional medical advice.

• Developed by: Bindupautra Jyotibrat (BJyotibrat)
• Shared by: BJyotibrat
• Model type: Causal language model, fine-tuned with LoRA
• Language(s) (NLP): English
• License: Llama 3.1 Community License
• Finetuned from model: meta-llama/Meta-Llama-3.1-8B-Instruct

Model Sources

• Repository: GitHub Repository
• Demo: BotMed Official Website

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

Intended for medical chatbot / medical question-answering conversational use cases.

Downstream Use

We envision several potential downstream uses for Llama-3.1-8B-Instruct-BotMed-v2 that extend beyond direct conversational question answering. These potential applications include:

(1) Retrieval-Augmented Generation (RAG) pipelines, serving as a domain-adapted generator integrated with verified medical knowledge bases,

(2) medical education and training, simulating doctor-patient interactions for clinical communication practice,

(3) synthetic data generation, creating synthetic dialogue datasets in conjunction with an evaluator model, and

(4) preliminary query drafting, acting as an assistive tool to generate initial responses for human-in-the-loop healthcare workflows.

(Note: This model is built with Llama and distributed under the Llama 3.1 Community License. Any downstream applications, derivative models, or software integrating this model must comply with Meta's Acceptable Use Policy and include the required "Built with Llama" attribution.)

---

Out-of-scope uses

We caution potential users of the model that Llama-3.1-8B-Instruct-BotMed-v2 is not a certified medical diagnostic tool, unlike verified clinical software. As such, it is completely unsafe and inappropriate for end-users that require actual medical advice, diagnosis, or treatment, for which a licensed human medical professional (such as a doctor or physician) would be the only appropriate resource.

Additionally, any use of this model must strictly adhere to the Llama 3.1 Acceptable Use Policy. Use cases that violate this policy, or commercial applications exceeding 700 million monthly active users without explicit permission from Meta, are strictly out of scope.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

• Fabricated attribution: Because the training data (ruslanmv/ai-medical-chatbot) consists of real doctor-patient Q&A transcripts, the model has learned to sign off responses with fabricated doctor names, specialties, and closing remarks (e.g. "Regards, Dr S.R.Raveendran, Sexologist"). These are stylistic artifacts inherited from training data, not real credentialed input, and should never be presented to end-users as if they came from an actual licensed physician.
• Lexical vs. semantic accuracy gap: Automated evaluation on a 100-question held-out test set shows relatively low lexical overlap with reference answers (F1: 0.1142, ROUGE-1: 0.1834, ROUGE-L: 0.1060) but moderate-to-high semantic similarity (Cosine Similarity: 0.5199, BERTScore F1: 0.7415). This indicates the model tends to phrase answers differently from the reference dataset while still capturing related medical concepts — it should not be assumed that semantically-similar output is clinically equivalent to the reference answer.
• No clinical validation: Evaluation metrics used (F1, ROUGE, Cosine Similarity, BERTScore) measure textual/semantic similarity to reference answers, not medical correctness. The model has not been evaluated by medical professionals or validated against clinical guidelines.
• Dataset-driven bias: The model's knowledge and tone are shaped entirely by the ruslanmv/ai-medical-chatbot dataset, which may not reflect current medical guidelines, may skew toward specific conditions/specialties represented in that dataset, and may contain its own biases or inaccuracies.
• Hallucination risk: As with any LLM, the model can generate plausible-sounding but incorrect or incomplete medical information.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. In particular:

• Do not present model-generated doctor names, credentials, or signatures as genuine.
• Always pair model output with human review, especially in any workflow that reaches an end-user seeking health guidance.
• Treat outputs as a drafting/assistive aid, not a source of medical truth.
• Consider domain-specific fact-checking or RAG grounding against verified medical sources before deploying in production.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "BJyotibrat/Llama-3.1-8B-Instruct-BotMed-v2"  # full merged model (base + LoRA already merged)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    dtype=torch.float16,
)

model.eval()

prompts = [
    "What are the symptoms of diabetes?",
    "How can high blood pressure be managed?"
]

for idx, t in enumerate(prompts, start=1):
    print(f"\nPrompt {idx}: {t}\n")

    messages = [{"role": "user", "content": t}]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

    inputs = tokenizer(
        prompt,
        return_tensors="pt",
        truncation=True,
        max_length=2048
    ).to("cuda")

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            repetition_penalty=1.2,
            pad_token_id=tokenizer.eos_token_id
        )

    input_length = inputs["input_ids"].shape[1]
    answer = tokenizer.decode(
        outputs[0][input_length:],
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False
    ).strip()

    print(f"Answer {idx}: {answer}\n")

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

Fine-tuned on the ruslanmv/ai-medical-chatbot dataset. The dataset was split into train/test sets using a 95/5 split (test_size=0.05, seed=65).

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Fine-tuned using LoRA (Low-Rank Adaptation) on Kaggle GPUs, with the base model loaded in 4-bit quantization via BitsAndBytes.

Preprocessing

Quantization (BitsAndBytes):

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation=attn_implementation,
    dtype=torch.float16,
)

model = prepare_model_for_kbit_training(model)

LoRA Configuration:

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

Training Hyperparameters

• Training regime: fp16=False, bf16=False (mixed-precision compute via bnb_4bit_compute_dtype=torch.float16)
• Batch size: 2 per device (train and eval), with gradient accumulation steps of 4
• Optimizer: paged_adamw_8bit
• Epochs: up to 3, with early stopping (added on top of load_best_model_at_end=True, metric_for_best_model="eval_loss") — training was manually stopped after the 2nd epoch showed no further improvement
• Evaluation/Save strategy: every 200 steps
• Logging: every 50 steps
• Warmup steps: 36
• Learning rate: 1e-4, cosine scheduler
• Gradient checkpointing: enabled
• Checkpoints kept: 3 (save_total_limit=3)
• Experiment tracking: Weights & Biases (wandb)

training_arguments = SFTConfig(
    output_dir=new_model,

    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    gradient_accumulation_steps=4,

    optim="paged_adamw_8bit",
    num_train_epochs=3,              # 3 epochs max, early stopping will protect

    eval_strategy="steps",
    eval_steps=200,
    save_strategy="steps",
    save_steps=200,

    logging_steps=50,

    warmup_steps=36,
    learning_rate=1e-4,
    lr_scheduler_type="cosine",

    fp16=False,
    bf16=False,

    report_to="wandb",
    gradient_checkpointing=True,

    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,

    save_total_limit=3,              # keep one checkpoint per epoch
)

Speeds, Sizes, Times

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

Final Weights & Biases run summary (best checkpoint, selected on lowest eval_loss):

Full step-by-step run history is available as a CSV export: wandb_run_history - BotMed v2.csv.

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

A held-out set of 100 medical questions, with model-generated answers compared against reference answers. The full question set, generated answers, and per-question scoring are available here: BotMed_v2_QA_Results.csv.

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

Evaluation was conducted on general open-domain medical Q&A prompts; no disaggregation by medical specialty, demographic, or question type was performed.

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

• F1 Score — token-level overlap between generated and reference answers.
• ROUGE-1 / ROUGE-L — unigram and longest-common-subsequence overlap, measuring lexical similarity to reference answers.
• Cosine Similarity — embedding-based similarity between generated and reference answers.
• BERTScore (Precision / Recall / F1) — contextual embedding similarity (via distilbert-base-uncased), capturing semantic alignment even when wording differs.

Results

Full per-question predictions and scores (100 questions) are available in BotMed_v2_QA_Results.csv.

Summary

Lexical overlap metrics (F1, ROUGE) are low, indicating the model rarely reproduces the reference answers' exact wording. However, BERTScore and Cosine Similarity indicate moderate-to-strong semantic alignment, suggesting the model generally captures the intent and relevant medical concepts of the reference answers while phrasing them differently. This is consistent with the free-form, conversational nature of the training data.

Example Outputs

Prompt 1: What are the symptoms of diabetes?

Answer 1: Symptoms include increased thirst and urination, blurring or double vision, fatigue, slow healing cuts/sores, etc. Also, if you have a family history of diabetes then it would be good to get yourself tested as well for early detection and treatment. You can also visit your doctor with these details so that he/she may advise accordingly.

Prompt 2: How can high blood pressure be managed?

Answer 2: High BP needs to be controlled by medications such as diuretics, calcium channel blockers like amlodipine or ACE inhibitors like enalapril. Lifestyle modification includes dietary restrictions, avoidance of salt in diet, regular exercise, etc.

(Note: raw model outputs include fabricated sign-offs such as doctor names and specialties)

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: Kaggle GPUs
• Hours used: 8 hours
• Cloud Provider: Kaggle

Technical Specifications

Model Architecture and Objective

Llama 3.1 8B Instruct architecture, fine-tuned via LoRA (rank 16, alpha 32) targeting the q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj modules, for medical chatbot dialogue generation. Base model loaded in 4-bit NF4 quantization for training.

Compute Infrastructure

Kaggle GPUs

Hardware

Kaggle GPU instance(s).

Software

🤗 Transformers, PEFT (LoRA), TRL (SFTConfig/SFTTrainer), BitsAndBytes, Weights & Biases and Kaggle.

Authors

Bindupautra Jyotibrat (BJyotibrat)

Contact

Email: [email protected]