dhivehi-en-qwen3-4b-lora-sentence

Qwen3-4B Dhivehi→English (sentence-level)

Qwen3-4B (decoder-only) + LoRA adapter for Dhivehi→English translation, trained on a sentence-level Dhivehi–English corpus (machine-translated).

Sentence-level variant. For multi-sentence input, see the paragraph models: Qwen-para / mT5-para. Suited for single-sentence input.

Scores (chrF / chrF++ / BLEU)

chrF is the metric to trust for Thaana; BLEU is unreliable (word segmentation / morphology).

Example

Input (dv): އެއީ، މިދިޔަ އަހަރުގެ މި މުއްދަތާ ބަލާއިރު، 7.6 އިންސައްތައިގެ ކުރިއެރުމެއް ކަމަށްވާއިރު، ދުވާލަކަށް 7،778 ފަތުރުވެރިން ރާއްޖެ ޒިޔާރަތްކުރެއެވެ.

Output (en): That is a 7.6 percent increase compared to the same period last year; on average, 7,778 tourists visit the Maldives per day.

Real held-out sample and this model's own output.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base, adapter = "Qwen/Qwen3-4B", "Neobe/dhivehi-en-qwen3-4b-lora-sentence"
tok = AutoTokenizer.from_pretrained(adapter)
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="cuda"),
    adapter).eval()

src = "ދިވެހިރާއްޖޭގެ ރައީސް މިއަދު ކެބިނެޓާ ބައްދަލުކުރެއްވި އެވެ."
msgs = [{"role":"user","content":
    "Translate the following Dhivehi text to English. Output only the translation, "
    f"no explanations.\n\nDhivehi: {src}\nEnglish:"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inp = tok(prompt, return_tensors="pt", truncation=True, max_length=3072).to("cuda")
out = model.generate(**inp, max_new_tokens=512, num_beams=1, repetition_penalty=1.15, no_repeat_ngram_size=3, do_sample=False)
print(tok.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True))

~9–10 GB VRAM (bf16). Qwen tokenizes Thaana at ~1.8 tokens/char — don't over-truncate the input.

Training

Base Qwen/Qwen3-4B; LoRA r=16, α=32, targets q/k/v/o+gate/up/down; bf16; adamw_torch LR 2e-4 cosine; max_length 1536; 1 epoch; effective batch ~32; gradient checkpointing.

Limitations

Domain = Maldivian news / press / Wikipedia; technical or informal English is out of distribution. Non-human references are machine-generated (distillation).

Citation

@misc{neobe_dhivehi_en_qwen3_4b_lora_sentence_2026,
  title  = {Qwen3-4B Dhivehi→English (sentence-level)},
  author = {Neobe},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Neobe/dhivehi-en-qwen3-4b-lora-sentence}}
}