zomi-qlora-v1

Zomi QLoRA v1 — Zomi Language Model

A Qwen 2.5 3B fine-tuned on Zomi (Zou) language text using QLoRA (rank 128).

This model can generate and understand Zomi text. It is the first publicly available language model for the Zomi language (spoken by ~500,000 people across Myanmar, India, and Bangladesh).

What This Model Can Do

• Generate coherent Zomi text
• Complete Zomi sentences and passages
• Understand Zomi grammar, vocabulary, and structure
• Serve as a foundation for further fine-tuning

What This Model Cannot Do (Yet)

• Follow instructions or commands (Phase 2 — instruction fine-tuning — is in progress)
• Translate between Zomi and English on demand
• Hold conversations

This is a continued pretraining / language adaptation model. It speaks Zomi but does not yet follow directions. Phase 2 (instruction tuning) will add command-following ability.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "paumkim/zomi-qlora-v1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype="auto",
)

# Generate Zomi text
prompt = "Pasian in hong it"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Fine-Tune It Yourself

This model is a QLoRA adapter (rank 128) on top of Qwen 2.5 3B. You can load it and fine-tune further:

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B", device_map="auto")
model = PeftModel.from_pretrained(base, model_name)
# Continue training with your own Zomi data

Training Details

Dataset

The model was trained on a corpus of Zomi text collected from public sources including:

• Zomi literature and publications
• Community-contributed content
• Public domain religious texts in Zomi

All data was collected with respect for community ownership of the Zomi language.

Why This Model Exists

Most AI models do not support Zomi. The Zomi language has ~500,000 speakers but zero commercial AI representation. This model is an attempt to change that — built by a Zomi speaker, for Zomi speakers.

Phase 2 (Coming)

Instruction fine-tuning is in progress. The Phase 2 model will follow commands in Zomi, enabling translation,问答, content generation, and more.

License

MIT — free to use, modify, and distribute. We believe the Zomi language should belong to its community, not to any company.