qwen3_bangla_lora

qwen3-bangla (LoRA adapters)

PEFT LoRA adapters for a Bangla (Bengali) instruction-tuned Qwen3-4B-Base. Load these on top of Qwen/Qwen3-4B-Base to get the Bangla instruction-following model.

This is SFT-only (no continual pre-training stage). The adapters were produced by supervised fine-tuning on the ihumaunkabir/alpaca-gpt4-bangla instruction dataset, using Unsloth.

Looking for a ready-to-run model? The merged + quantized GGUF (no base model needed) lives at ihumaunkabir/qwen3_bangla_q4_k_m_gguf.

embed_tokens and lm_head are included to help the model adapt its vocabulary to Bengali script (the base Qwen3-4B model has a ~152K multilingual vocabulary, but the Bangla portion benefits from SFT-time adaptation).

---

Training

Stage 1 -- Supervised Fine-Tuning (SFT)

---

How to use

Load adapters with Unsloth (recommended, 2x faster inference)

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name = "ihumaunkabir/qwen3_bangla_lora",
    max_seq_length = 2048,
    load_in_4bit = True,
)
FastModel.for_inference(model)

alpaca_prompt = """নিচে একটি নির্দেশনা দেওয়া আছে, যা একটি কাজের বর্ণনা দেয়। অনুরোধটি যথাযথভাবে সম্পূর্ণ করে একটি উত্তর লিখুন।

### নির্দেশনা:
{}

### উত্তর:
{}"""

inputs = tokenizer([alpaca_prompt.format("বাংলাদেশের রাজধানীর নাম লেখো।", "")], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, use_cache=True)
print(tokenizer.batch_decode(outputs))

Note: this uses Unsloth's FastModel API (not FastLanguageModel). FastModel is the unified entry point for newer model families including Qwen3.

Load with PEFT (no Unsloth)

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained(
    "ihumaunkabir/qwen3_bangla_lora",
    load_in_4bit = True,
)
tokenizer = AutoTokenizer.from_pretrained("ihumaunkabir/qwen3_bangla_lora")

Limitations

1. Smoke-test training volume (max_steps=120) -- ~1,920 samples. Increase for production.
2. No safety alignment. No RLHF, DPO, or red-teaming.
3. QLoRA 4-bit during training can slightly degrade quality vs. bf16.
4. Requires the base model to be loaded separately -- this is an adapter, not a standalone model.
5. Alpaca prompt format required. The model will produce poor output if prompted with a different template (e.g. ChatML).

---

Citation

If you use these adapters, please cite both this repo and the alpaca-gpt4-bangla dataset, plus the base Qwen3 model.

This repo (LoRA adapters)

@misc{qwen3-bangla-lora,
  author       = {ihumaunkabir},
  title        = {qwen3-bangla: Bangla SFT LoRA adapters for Qwen3-4B-Base},
  year         = {2026},
  url          = {https://huggingface.co/ihumaunkabir/qwen3_bangla_lora},
  note         = {SFT on alpaca-gpt4-bangla, trained with Unsloth}
}

SFT dataset -- alpaca-gpt4-bangla

@misc{alpaca-gpt4-bangla,
  author       = {ihumaunkabir},
  title        = {alpaca-gpt4-bangla: A Bangla instruction-following dataset},
  year         = {2026},
  url          = {https://huggingface.co/datasets/ihumaunkabir/alpaca-gpt4-bangla},
  note         = {Machine translation (Korean -> Bangla) of FreedomIntelligence/alpaca-gpt4-korean}
}

Base model

@misc{qwen3,
  author       = {Qwen Team},
  title        = {Qwen3-4B-Base},
  year         = {2025},
  url          = {https://huggingface.co/Qwen/Qwen3-4B-Base}
}