Gemma_3_4B

Gemma 3 4B — Bilingual (Amharic + English) Medical QA

Fine-tuned LoRA adapters on top of google/gemma-3-4b-it for medical question-answering in Amharic and English. This is the lightweight counterpart to the 12B model, trained on the same bilingual medical QA dataset and intended for lower-cost inference.

⚠️ Medical disclaimer. This model is a research prototype. Its outputs are AI-generated and may be incomplete or incorrect. It is not a medical device and must not be used for diagnosis or treatment decisions. Always consult a qualified healthcare professional. Outputs should be reviewed by a clinician before any downstream use.

---

Model details

LoRA configuration

Training configuration

---

Dataset

Trained on a bilingual medical QA dataset with parallel English and Amharic question–answer pairs. Each source row contributes two training examples (one English pair, one Amharic pair).

Dataset provenance: content compiled from public health websites/articles, medical textbooks and guidelines, expert/clinician-written material, and an existing medical QA dataset. Amharic produced by professional translation. Medical content reviewed by a qualified health professional. See the dataset card for full details.

---

Training results

Training loss trajectory

Training loss dropped sharply and continued to decline through training:

Per-epoch eval progress

Eval loss reached its minimum at epoch 2 (1.3772) and rose slightly at epoch 3 (1.4402), while training loss continued to fall — indicating the onset of mild overfitting in the final epoch. The end-of-epoch-2 checkpoint (checkpoint-576) may generalize slightly better than the final checkpoint. The final adapter at the repo root corresponds to epoch 3 (checkpoint-864); users who prefer the best-validation checkpoint can load checkpoint-576.

Comparison with the 12B model

For reference, the larger Gemma 3 12B version trained on the same data reached a best eval loss of 1.224 and accuracy 0.696. This 4B model trades some quality for substantially smaller size and lower inference cost.

Checkpoints

---

How to use

These are LoRA adapters — load them on top of the base model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "google/gemma-3-4b-it"
ADAPTER = "Walelign/Gemma_3_4B"

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

question = "የስኳር በሽታ ምልክቶች ምንድን ናቸው?"
msgs = [{"role": "user", "content": question}]
enc = tok.apply_chat_template(
    msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

with torch.no_grad():
    out = model.generate(**enc, max_new_tokens=256, do_sample=False)

print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

Requirements

torch  (CUDA build matching your driver)
transformers>=4.50.0
peft
accelerate
bitsandbytes   # only if loading in 4-bit

---

Evaluation examples

Qualitatively tested on Amharic STI-related medical questions. Example questions:

1. በግብረ ሥጋ ግንኙነት የሚተላለፉ በሽታዎች ምንድን ናቸው?
2. የግብረ ሥጋ ግንኙነት በሽታዎችን እንዴት መከላከል ይቻላል?
3. የ ኤች አይ ቪ (HIV) ምልክቶች ምንድን ናቸው?
4. የግብረ ሥጋ ግንኙነት በሽታ እንዳለብኝ እንዴት ማወቅ እችላለሁ?
5. ጨብጥ (ጨብጥ በሽታ) እንዴት ይታከማል?
6. ቂጥኝ (ሲፊሊስ) ምንድን ነው እና ምልክቶቹ ምንድን ናቸው?
7. ኮንዶም መጠቀም ከበሽታ ይከላከላል?
8. የግብረ ሥጋ ግንኙነት በሽታ ምርመራ የት ማድረግ እችላለሁ?
9. የግብረ ሥጋ ግንኙነት በሽታዎች ካልታከሙ ምን ችግር ያስከትላሉ?
10. ነፍሰ ጡር ሴት የግብረ ሥጋ ግንኙነት በሽታ ቢኖራት በልጁ ላይ ተጽዕኖ ይኖረዋል?

Full test outputs are provided in sti_qa_results_4b.txt.

---

Limitations and risks

• Training data was clinician-reviewed, but model outputs are not. The dataset was reviewed by a health professional; the model can still generate answers that differ from its training data, so its generated outputs have not been individually validated and may be incorrect.
• Smaller model, lower ceiling. At 4B parameters this model is less capable than the 12B version; expect somewhat weaker reasoning and fluency, especially on complex or rare medical questions.
• Domain-narrow. Medical knowledge comes primarily from the small fine-tuning dataset (~2,300 pairs).
• Treatment/dosage caution. Answers about treatment, medication, or pregnancy should be treated with particular caution.
• Inherits base-model limitations and biases from google/gemma-3-4b-it.

Intended use

Research and educational exploration of bilingual (Amharic/English) medical QA, and lower-cost deployment where the 12B model is too expensive to serve. Not for clinical, diagnostic, or treatment use.

License

Derived from google/gemma-3-4b-it and subject to the Gemma Terms of Use.

Citation

@misc{sewunetie_tilahun_gemma3_4b_amharic_medqa,
  title  = {Gemma 3 4B Bilingual Amharic-English Medical QA (LoRA)},
  author = {Sewunetie, Walelign Tewabe and Tilahun, Surafel L.},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Walelign/Gemma_3_4B}}
}