Downloads · 30 days
27
3% of all-time downloads
cngchis/Llama3.1-medical-GGUF
Llama3.1-medical-GGUF is a question answering model from cngchis. Use it when the input is a question plus a passage. The card lists the license as mit.
Fine-tuned Meta-Llama-3.1-8B-Instruct on a Vietnamese medical QA dataset using LoRA + Unsloth. Quantized to GGUF format for local inference via llama.cpp / Ollama.
Downloads · 30 days
27
3% of all-time downloads
All-time downloads
799
Public
Repo size
49.1 GB
Likes
0
Public
Click a slice to open those files.
.gguf49.1 GB · 100%
From the Hugging Face model README
Fine-tuned Meta-Llama-3.1-8B-Instruct on a Vietnamese medical QA dataset using LoRA + Unsloth. Quantized to GGUF format for local inference via llama.cpp / Ollama.
| Base model | meta-llama/Meta-Llama-3.1-8B-Instruct-bnb-4bit |
| Fine-tuning method | LoRA (r=16, alpha=16, RSLoRA) |
| Training framework | Unsloth + TRL SFTTrainer |
| Dataset | Vietnamese Medical QA — 9,000 samples |
| Training hardware | Tesla V100-SXM2 32GB |
| Language | Vietnamese 🇻🇳 |
| Domain | Medical QA (symptoms, diagnosis, medication) |
Evaluated on held-out test set of 931 samples.
| Metric | Score |
|---|---|
| BLEU-4 (char-level) | |
| ROUGE-L | |
| BERTScore F1 | |
| Perplexity (mean) | |
| Inference latency (p95) |
| Filename | Quant | Size | Recommended for |
|---|---|---|---|
medical-llama3.1-8b-q2_k.gguf | Q2_K | ~3.0 GB | RAM < 4GB, quick test |
medical-llama3.1-8b-q3_k_m.gguf | Q3_K_M | ~3.9 GB | CPU-only, lightweight |
medical-llama3.1-8b-q4_k_m.gguf | Q4_K_M | ~4.8 GB | ✅ Recommended — Ollama / production |
medical-llama3.1-8b-q5_k_m.gguf | Q5_K_M | ~5.5 GB | High quality, small GPU |
medical-llama3.1-8b-q6_k.gguf | Q6_K | ~6.1 GB | Near fp16 quality |
medical-llama3.1-8b-q8_0.gguf | Q8_0 | ~8.5 GB | Best quality, benchmark |
./llama.cpp/build/bin/llama-cli \
-m medical-llama3.1-8b-q4_k_m.gguf \
-p "You are a customer care doctor. Be polite and answer all questions from the customer." \
--prompt "Chào bác sĩ, em bị đau đầu và sốt nhẹ thì có cần đi khám không ạ?" \
-n 256
# Create Modelfile
cat > Modelfile << 'EOF'
FROM medical-llama3.1-8b-q4_k_m.gguf
SYSTEM "You are a customer care doctor. Be polite and answer all questions from the customer."
EOF
ollama create vietnam-medical -f Modelfile
ollama run vietnam-medical "Chào bác sĩ, em bị đau đầu và sốt nhẹ thì có cần đi khám không ạ?"
from llama_cpp import Llama
llm = Llama(
model_path="medical-llama3.1-8b-q4_k_m.gguf",
n_ctx=1024,
n_gpu_layers=-1, # offload all layers to GPU if available
)
response = llm.create_chat_completion(
messages=[
{
"role": "system",
"content": "You are a customer care doctor. Be polite and answer all questions from the customer."
},
{
"role": "user",
"content": "Chào bác sĩ, em bị đau đầu và sốt nhẹ thì có cần đi khám không ạ?"
}
],
max_tokens=256,
)
print(response["choices"][0]["message"]["content"])
r = 16
lora_alpha = 16
lora_dropout = 0.5
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"]
use_rslora = False
per_device_train_batch_size = 2
gradient_accumulation_steps = 4
num_train_epochs = 3
learning_rate = 2e-4
lr_scheduler_type = "linear"
max_seq_length = 1024
optim = "adamw_8bit"
⚠️ This model is intended for informational purposes only and should not be used as a substitute for professional medical advice, diagnosis, or treatment. Always consult a qualified healthcare provider.
This model is built on Meta Llama 3.1 and is subject to the Llama 3.1 Community License Agreement.