ChatgaiyyaLM-0.5B

[!IMPORTANT] Superseded. Use ChatgaiyyaLM-0.5B-B2_subset_ctg_syl_bn instead — 65.64 chrF on Chittagonian→Bangla against this adapter's 35.66, trained jointly with Sylheti and standard Bangla and uploaded without the lossy repackaging step described below.

ChatgaiyyaLM 0.5B — Chittagonian adapter

LoRA adapter adapting Qwen/Qwen2.5-0.5B to Chittagonian (চাটগাঁইয়া), a Bangla dialect spoken by tens of millions of people with almost no NLP resources.

• Benchmark & corpus: ChatgaiyyaBench
• Try the rule system: ChatgaiyyaBridge Space
• Code & full results: https://github.com/A-K-M-Asifuzzaman/Chatgaiya-AI

Quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

REPO = "AsifZaman1912/ChatgaiyyaLM-0.5B"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B", dtype=torch.float16)
model.resize_token_embeddings(len(tok))       # REQUIRED: adapter adds 3,914 tokens
model = PeftModel.from_pretrained(model, REPO).eval()

prompt = "চাটগাঁইয়া থেকে প্রমিত বাংলায় অনুবাদ করো:\nঅ্যাঁই ভাত ন হাইয়্যুম\nঅনুবাদ:"
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Load the tokenizer from this repo. Token ids were assigned in insertion order, so a tokenizer rebuilt from elsewhere silently misaligns every added token.

Measured performance — read this carefully

chrF on the gold translation track of ChatgaiyyaBench, 120 fixed-seed items per direction.

The last two rows are the honest part.

1. The training run reported 58.46 / 52.63. Re-measuring these published weights locally gave 35.66 / 24.10. The weights here are packaged from a size-reduced archive of that run, and the round trip loses accuracy for a reason I was unable to isolate — I ruled out tokenizer misalignment (3,914 tokens, zero id mismatches), fp16 precision (fp32 scored identically), and a mis-targeted embedding freeze. The numbers in bold are what these files actually do.
2. A rule-based transducer with no GPU cost beats every neural arm at Bangla → Chittagonian. That is a real finding about the task, not an omission.

The adapter still clears both meaningful baselines — the unadapted model (20.54) and copying the input (29.51) — so it is a working dialect adapter, just not as strong as the training run suggested. A re-trained release without the lossy repackaging step is in progress.

Training

Vocabulary augmentation cuts Chittagonian fertility from 5.722 to 2.254 tokens/word. Across five tokenizers, the more Bangla-specialised the tokenizer, the larger its dialect penalty — banglat5 has the best absolute Bangla fertility (1.054) and the worst dialect tax (+72.6%).

Limitations

• 0.5B parameters. Qwen2.5-1.5B needs ~14.5 GB before activations under PEFT's duplicated embeddings and OOMs a 16 GB T4.
• Fixed instruction-tuning budget, not full epochs.
• Four of eight benchmark tracks are silver (auto-derived labels or transduced text), awaiting native-speaker validation.
• Research artifact, not a production translator. For Bangla → Chittagonian specifically, the rule transducer is better.

Citation

@software{chatgaiyyalm,
  title  = {ChatgaiyyaLM: Language Adaptation and Benchmarking for Chittagonian},
  author = {Asifuzzaman, A. K. M.},
  year   = {2026},
  url    = {https://github.com/A-K-M-Asifuzzaman/Chatgaiya-AI}
}