Llama-3.2-3B-Instruct-Mongolian

Llama-3.2-3B-Instruct-Mongolian

A LoRA fine-tuned version of meta-llama/Llama-3.2-3B-Instruct for Mongolian language instruction-following and chat.

Model Description

This model adapts Llama 3.2 3B Instruct to understand and generate fluent Mongolian text. It was fine-tuned using LoRA (Low-Rank Adaptation) on the saillab/alpaca-mongolian-cleaned dataset containing ~41,600 Mongolian instruction-following examples.

The base model struggles with Mongolian, producing garbled or incoherent text. After fine-tuning, the model generates fluent, coherent Mongolian responses across a wide range of topics.

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

BASE_MODEL = "meta-llama/Llama-3.2-3B-Instruct"
ADAPTER = "munkhbayar-batkhuu/Llama-3.2-3B-Instruct-Mongolian"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, torch_dtype=torch.float16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

messages = [
    {"role": "system", "content": "You are a helpful assistant that responds in Mongolian."},
    {"role": "user", "content": "Монгол улсын нийслэл хаана байдаг вэ?"},
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9, do_sample=True)

response = tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)
# Output: Монгол улсын нийслэл нь Улаанбаатар хот юм.

Training Details

Chat Template

Training used the Llama 3.2 chat template with system prompt: "You are a helpful assistant that responds in Mongolian."

Benchmark Results

MM-Eval (Mongolian Multi-task Evaluation)

MM-Eval (arXiv:2411.09492) is a hierarchical benchmark for evaluating LLMs on Mongolian language tasks across 1,840 items.

Perplexity (on eval split, 2,081 samples)

BLEU / ROUGE-L (200 eval samples)

Example Outputs

Prompt: "Монгол улсын нийслэл хаана байдаг вэ?" (Where is the capital of Mongolia?)

Prompt: "Хүүхдэд зориулж богино үлгэр бичнэ үү." (Write a short story for children.)

Prompt: "Монгол хоолны жор бичнэ үү." (Write a Mongolian food recipe.)

Limitations

• Domain: Trained on general instruction-following data; may not perform well on specialized domains (medical, legal, technical)
• Math/Reasoning: Mathematical reasoning did not improve (slightly declined on MM-Eval reasoning)
• Hallucination: Like all LLMs, may generate plausible but factually incorrect information
• Sequence Length: Trained with max 512 tokens; may degrade on longer inputs
• Model Size: 3B parameters -- larger models would likely achieve better results

Framework Versions

• PEFT: 0.18.1
• TRL: 0.27.2
• Transformers: 5.1.0
• PyTorch: 2.10.0+cu128
• Datasets: 4.5.0

Citation

If you use this model, please cite:

@misc{llama32-3b-mongolian-2026,
  title={Llama-3.2-3B-Instruct-Mongolian},
  author={Munkhbayar Batkhuu},
  year={2026},
  url={https://huggingface.co/munkhbayar-batkhuu/Llama-3.2-3B-Instruct-Mongolian}
}

Acknowledgments

• Meta AI for the Llama 3.2 base model
• SAIL Lab for the Mongolian Alpaca dataset
• MM-Eval authors for the Mongolian evaluation benchmark