Llama-3.2-3B-Instruct-Medical-Finetuned-merged-GGUF

Llama-3.2-3B-Instruct-Medical-Finetuned-merged GGUF

GGUF quantizations of MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetuned-merged — a 3.21B-parameter Llama-3.2 model adapted to the medical domain using QLoRA, with the LoRA weights merged back into the base.

Quantized by tinyopsec.

---

Model Details

---

About the Original Model

Med-LLaMA3.2-3B is the 3B (balanced / mid-tier) member of the Med-LLaMA3 family introduced in the paper "Med-LLaMA3: Advancing Medical Question-Answering Through Parameter-Efficient Fine-Tuning of Large Language Models" (Applied Sciences, 2026). The family adapts the LLaMA-3 architecture to the medical domain by training only a small fraction of the base model's parameters (5.70% for this 3B variant), achieving strong medical question-answering performance while keeping the memory footprint low.

Training Details

• Base model: meta-llama/Llama-3.2-3B-Instruct
• Method: QLoRA fine-tuning (4-bit NF4 base + LoRA, r=128, α=256, all linear layers)
• Trainable parameters: 194.51 M = 5.70% of the 3.40 B total
• Merging: merge_and_unload() into the base
• Released weights: bfloat16 (full precision; not quantized)
• Training datasets: MohamedAhmedAE/Med_LLaMa3_fine-tuning_dataset, medalpaca/medical_meadow_medqa, medalpaca/medical_meadow_medical_flashcards, medalpaca/medical_meadow_wikidoc, medalpaca/medical_meadow_wikidoc_patient_information, medalpaca/medical_meadow_cord19, medalpaca/medical_meadow_pubmed_causal, openlifescienceai/medmcqa, bigbio/med_qa, qiaojin/PubMedQA, deepset/covid_qa_deepset

This repository contains the merged standalone model. No adapter or PEFT is required for inference.

---

Available Quantizations

---

VRAM Requirements

Approximate total VRAM required for inference (weights + context overhead).

---

Usage

llama.cpp

./llama-cli -m model_q4_k_m.gguf \
  -p "You are a helpful medical assistant. Answer the following question accurately and concisely.\n\nQuestion:\nWhat are the early symptoms of type 2 diabetes?\n\nAnswer:\n" \
  -n 512 --temp 0.0

llama-cpp-python

from llama_cpp import Llama

llm = Llama(model_path="model_q4_k_m.gguf", n_ctx=4096)

output = llm(
    "You are a helpful medical assistant. Answer the following question accurately and concisely.\n\nQuestion:\nWhat is the mechanism of action of metformin?\n\nAnswer:\n",
    max_tokens=512,
    temperature=0.0,
)
print(output["choices"][0]["text"])

LM Studio

1. Download any .gguf file from this repository
2. Open LM Studio → Load Model → select the file
3. Use the prompt format below in the chat or playground

Ollama

ollama run hf.co/tinyopsec/Llama-3.2-3B-Instruct-Medical-Finetuned-merged-GGUF:Q4_K_M

---

Prompt Format

For medical question answering, use the standard Llama 3.2 chat template:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful medical assistant. Answer the user's question accurately and concisely.<|eot_id|><|start_header_id|>user<|end_header_id|>

{your question here}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

For general instruction use, the default chat template also works.

---

Intended Uses

Primary Use Cases

• Medical question answering (multiple-choice and open-ended)
• Clinical knowledge lookup and clinical decision support assistance
• A balanced mid-tier option when the 1B is too small and the 8B is too heavy
• A research baseline for parameter-efficient fine-tuning of small LLaMA models in healthcare

Out of Scope / Not Intended For

• Autonomous clinical decision-making or direct patient care without a qualified clinician in the loop
• Generating definitive diagnoses, prescriptions, or treatment plans
• Use as a substitute for professional medical advice, emergency services, or licensed care

---

Limitations

• No formal benchmark results are claimed in this release. External medical benchmarks should be run before production use.
• The model can produce fluent but incorrect medical information.
• Training data may contain biases or inaccuracies; outputs should be verified against authoritative sources.
• The model has not been established as safe for autonomous, high-impact, medical, financial, or legal action.
• Context length is large (128K), but very long inputs may degrade performance.
• Medical knowledge is not updated in real time; the model may not reflect the latest clinical guidelines.

---

License

The parent model is released under the Llama 3.2 Community License. Any use of this GGUF quantization must comply with the same license terms. Confirm the source dataset licenses before commercial deployment.

---

Original Model

• Parent model: MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetuned-merged
• Base model: meta-llama/Llama-3.2-3B-Instruct
• Paper: Med-LLaMA3: Advancing Medical Question-Answering Through Parameter-Efficient Fine-Tuning of Large Language Models (Applied Sciences, 2026)

Citation

@article{aboelenen2026medllama3,
  title={Med-LLaMA3: Advancing Medical Question-Answering Through Parameter-Efficient Fine-Tuning of Large Language Models},
  author={Abo El-Enen, Mohamed Ahmed and Ismail, Sally S. and Nazmy, Taymoor Mohamed},
  journal={Applied Sciences},
  year={2026}
}