Llama-3-8B-Instruct-BotMed-v1

BotMed/Llama-3-8B-Instruct-BotMed-v1

<!-- Provide a quick summary of what the model is/does. -->

BotMed-v1 is a fine-tuned version of Llama 3 8B Instruct, adapted for medical chatbot use cases using LoRA.

Model Description

<!-- Provide a longer summary of what this model is. -->

BotMed-v1 is a fine-tuned version of Meta's Llama 3 8B Instruct, specialized for medical chatbot and question-answering use cases. It was trained using LoRA (Low-Rank Adaptation) on the ruslanmv/ai-medical-chatbot dataset, with the base model loaded in 4-bit NF4 quantization to enable efficient fine-tuning on Kaggle GPUs.

The LoRA adapters (rank 8, alpha 16) target the attention and MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), allowing the model to adapt to medical dialogue patterns while preserving the general instruction-following capabilities of the base Llama 3 model. Training was run for 2 epochs, using a cosine learning rate schedule and the paged AdamW 8-bit optimizer.

This is the v1 release of BotMed and the first fine-tuning run in the project. It is not a certified diagnostic tool and should not be used as a substitute for professional medical advice.

• Developed by: Bindupautra Jyotibrat (BJyotibrat)
• Shared by: BJyotibrat
• Model type: Causal language model, fine-tuned with LoRA
• Language(s) (NLP): English
• License: Llama 3 Community License
• Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct

Model Sources

<!-- Provide the basic links for the model. -->

• Repository: GitHub Repository
• Demo: BotMed Official Website

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

Intended for medical chatbot / medical question-answering conversational use cases.

Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

As the first fine-tuning run in the BotMed project, Llama-3-8B-Instruct-BotMed-v1 is best suited to exploratory and internal downstream uses rather than production deployment. Potential applications include:

(1) baseline comparison, serving as a reference point against later BotMed versions (e.g. v2) to evaluate the impact of changes in LoRA configuration, epochs, or dataset handling,

(2) early-stage prototyping of medical chatbot interfaces and RAG pipelines before swapping in a more thoroughly evaluated model,

(3) internal experimentation with prompt design and fine-tuning workflows for medical dialogue generation, and

(4) further fine-tuning or continued training as a starting checkpoint for downstream medical NLP experiments.

(Note: This model is built with Llama and distributed under the Llama 3 Community License. Any downstream applications, derivative models, or software integrating this model must comply with Meta's Acceptable Use Policy and include the required "Built with Llama" attribution.)

Given the lack of formal evaluation for this version (see Bias, Risks, and Limitations), it is not recommended for downstream use cases that reach real end-users without further validation — prefer v2 for those.

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

Not a certified medical diagnostic tool. Unsafe and inappropriate for end-users seeking actual medical advice, diagnosis, or treatment — a licensed human medical professional should be the only resource used for that. Any use must comply with the Llama 3 Acceptable Use Policy.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

• Fabricated attribution: Because the training data (ruslanmv/ai-medical-chatbot) consists of real doctor-patient Q&A transcripts, the model has learned to sign off responses with fabricated doctor names, specialties, and closing remarks (e.g. "Regards, Dr. Shinas Hussain, General & Family Physician"). These are stylistic artifacts inherited from training data, not real credentialed input, and should never be presented to end-users as if they came from an actual licensed physician.
• No formal evaluation: Unlike v2, this v1 release has not been scored against held-out reference answers using metrics such as F1, ROUGE, Cosine Similarity, or BERTScore. Its quality has only been assessed qualitatively via example prompts.
• No clinical validation: The model has not been evaluated by medical professionals or validated against clinical guidelines.
• Dataset-driven bias: The model's knowledge and tone are shaped entirely by the ruslanmv/ai-medical-chatbot dataset, which may not reflect current medical guidelines and may skew toward specific conditions/specialties represented in that dataset.
• Hallucination risk: As with any LLM, the model can generate plausible-sounding but incorrect or incomplete medical information.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. In particular:

• Do not present model-generated doctor names, credentials, or signatures as genuine.
• Always pair model output with human review, especially in any workflow that reaches an end-user seeking health guidance.
• Treat outputs as a drafting/assistive aid, not a source of medical truth.
• Given the lack of formal evaluation for this version, prefer v2 for anything beyond experimentation, where possible.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "BJyotibrat/Llama-3-8B-Instruct-BotMed-v1"  # full merged model (base + LoRA already merged)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    dtype=torch.float16,
)

model.eval()

prompts = [
    "I have frequent fever and pain in chest"
]

for idx, t in enumerate(prompts, start=1):
    print(f"\nPrompt {idx}: {t}\n")

    messages = [{"role": "user", "content": t}]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

    inputs = tokenizer(
        prompt,
        return_tensors="pt",
        truncation=True,
        max_length=2048
    ).to("cuda")

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            repetition_penalty=1.2,
            pad_token_id=tokenizer.eos_token_id
        )

    input_length = inputs["input_ids"].shape[1]
    answer = tokenizer.decode(
        outputs[0][input_length:],
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False
    ).strip()

    print(f"Answer {idx}: {answer}\n")

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

Fine-tuned on the ruslanmv/ai-medical-chatbot dataset.

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Fine-tuned using LoRA (Low-Rank Adaptation) on Kaggle GPUs, with the base model loaded in 4-bit quantization via BitsAndBytes.

Preprocessing

Quantization (BitsAndBytes):

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation=attn_implementation,
    dtype=torch.float16,
)

model = prepare_model_for_kbit_training(model)

LoRA Configuration:

peft_config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.1,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

Training Hyperparameters

• Training regime: fp16=False, bf16=False (mixed-precision compute via bnb_4bit_compute_dtype=torch.float16)
• Batch size: 2 per device (train and eval), with gradient accumulation steps of 4
• Optimizer: paged_adamw_8bit
• Epochs: 2 configured, with early stopping (training was manually stopped once eval loss stopped improving, ending at ~1.52 epochs)
• Evaluation/Save strategy: every 200 steps
• Logging: every 50 steps
• Warmup steps: 36
• Learning rate: 1e-4, cosine scheduler
• Gradient checkpointing: enabled
• Checkpoints kept: 2 (save_total_limit=2)
• Experiment tracking: Weights & Biases (wandb)

training_arguments = SFTConfig(
    output_dir=new_model,

    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    gradient_accumulation_steps=4,

    optim="paged_adamw_8bit",
    num_train_epochs=2,

    eval_strategy="steps",
    eval_steps=200,
    save_strategy="steps",
    save_steps=200,

    logging_steps=50,

    warmup_steps=36,
    learning_rate=1e-4,
    lr_scheduler_type="cosine",

    fp16=False,
    bf16=False,

    report_to="wandb",
    gradient_checkpointing=True,

    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,

    save_total_limit=2,
)

Speeds, Sizes, Times

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

Final logged Weights & Biases metrics (last recorded checkpoint, global step 1800):

Full run history is available as a CSV export: wandb_run_history - BotMed v1.csv.

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

No formal held-out evaluation set has been scored for this version.

Results

No formal evaluation metrics (F1, ROUGE, Cosine Similarity, BERTScore) have been computed for this version.

Example Outputs

Prompt: I have frequent fever and pain in chest

Answer: Hello, I have gone through your query. From what you have mentioned, it seems that you are having a dry cough. I would suggest you to use some cough syrups like Tab Robitussin or Tab Mucodyne for 3-5 days. Also, you can use some expectorant like Tab Mucopain for 3-5 days. If you have a fever, then you can use some antipyretic like Tab Paracetamol for 3-5 days.

(Note: raw model outputs include fabricated sign-offs such as doctor names and specialties)

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: Kaggle GPUs
• Hours used: 6 hours
• Cloud Provider: Kaggle

Technical Specifications

Model Architecture and Objective

Llama 3 8B Instruct architecture, fine-tuned via LoRA (rank 8, alpha 16) targeting the q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj modules, for medical chatbot dialogue generation. Base model loaded in 4-bit NF4 quantization for training.

Compute Infrastructure

Kaggle GPUs

Hardware

Kaggle GPU instance(s).

Software

🤗 Transformers, PEFT (LoRA), TRL (SFTConfig/SFTTrainer), BitsAndBytes, Weights & Biases and Kaggle.

Authors

Bindupautra Jyotibrat (BJyotibrat)

Contact

Email: [email protected]