qwen2.5-7b-cantonese-lora-v3

Qwen2.5-7B-Instruct + Cantonese LoRA v3

LoRA adapter fine-tuned on Qwen2.5-7B-Instruct-4bit for natural Hong Kong Cantonese text generation.

Model description

This is a LoRA adapter that adds Cantonese style transfer capability to mlx-community/Qwen2.5-7B-Instruct-4bit. It was trained on 1500 long-phrase (50-200 character) parallel sentence pairs from the HKAllen/cantonese-chinese-parallel-corpus and raptorkwok/cantonese-chinese-parallel-corpus datasets.

The adapter covers all attention projections (q, k, v, o) and all MLP projections (gate, up, down) in the 28 transformer layers — a total of 196 LoRA modules with rank 16, alpha 32, dropout 0.05 (40.7M trainable params, 3.3% of base model).

Intended use

• In-domain: Casual Cantonese Q&A, short dialogue, descriptive text in Hong Kong Cantonese register
• Out-of-scope: General Chinese NLP, formal Mandarin writing, factual Q&A (use base Qwen2.5-7B-Instruct instead)

How to use (MLX)

import mlx.core as mx
import mlx.nn as nn
from mlx_lm import load, generate
from mlx.utils import tree_flatten, tree_unflatten

# Load base model
model, tokenizer = load("mlx-community/Qwen2.5-7B-Instruct-4bit")

# Wrap with LoRA (7 targets, matching training config)
LORA_TARGETS = ["q_proj", "k_proj", "v_proj", "o_proj",
                "gate_proj", "up_proj", "down_proj"]
LORA_CONFIG = {"rank": 16, "scale": 32, "dropout": 0.05}

for layer in model.layers:
    types = (nn.Linear, nn.QuantizedLinear)
    keys = set()
    for name, child in layer.named_modules():
        if isinstance(child, types):
            for t in LORA_TARGETS:
                if name.endswith(t) or f".{t}" in name:
                    keys.add(name); break
    if not keys: continue
    named_dict = dict(layer.named_modules())
    from mlx_lm.tuner.lora import LoRALinear
    lora_layers = []
    for k in keys:
        layer_obj = named_dict[k]
        if isinstance(layer_obj, nn.QuantizedLinear):
            lora_layers.append((k, LoRALinear.from_base(layer_obj, **LORA_CONFIG)))
        else:
            lora_layers.append((k, layer_obj.to_lora(**LORA_CONFIG)))
    layer.update_modules(tree_unflatten(lora_layers))

# Load LoRA weights
model.load_weights("adapters.safetensors", strict=False)

# Generate
prompt = "<|im_start|>system\nYou are a helpful assistant who speaks Cantonese.<|im_end|>\n<|im_start|>user\n你喺邊度出世?<|im_end|>\n<|im_start|>assistant\n"
output = generate(model, tokenizer, prompt=prompt, max_tokens=100, verbose=False)
print(output)

Training data

Cantonese particle density (per 97,514 response chars): 嘅 1.93%, 喺 0.73%, 咗 0.35%, 唔 0.33%, 嚟 0.16%, 嗰 0.14%.

Training procedure

Loss curve

Evaluation

20-prompt A/B generation (base vs tuned), same prompts across phases.

Qualitative findings:

✅ Long-form generation no longer collapses to short replies (Phase 2's mode collapse fixed for some prompts)
✅ Real Cantonese dialogue in extended responses
✅ Punctuation and tone markers in extended form
⚠️ List-mode repetition collapse in some prompts (e.g. "港式炒魚片" repeated)
⚠️ Occasional Mandarin-character bleed for rare proper nouns (e.g. "alertController")

See ab_results_v3.md in the repo for the full per-prompt comparison.

Limitations

1. Particle count is the wrong metric for long outputs. A 200-char Cantonese paragraph can be clearly Cantonese with 0 particles but full Cantonese syntax.
2. Mode collapse shifts but does not disappear. v2 had short-reply collapse; v3 has list-repetition collapse. DPO/ORPO with human preference data would likely improve.
3. Training data is parallel sentence pairs, not natural Cantonese conversations. The model learns sentence-level style, not conversational register.
4. Some Mandarin-Chinese proper nouns bleed through because the base model's tokenizer treats some characters as Mandarin.

Reproducibility

# Setup
uv venv ~/mlx-env --python 3.11
source ~/mlx-env/bin/activate
uv pip install mlx mlx-lm transformers peft datasets

# Pipeline
cd ~/llm-finetune
python scripts/prep_dataset_v3.py     # 1500 long samples
python scripts/dry_run_v3.py          # safety check
python scripts/train_v3.py            # 300 iters, ~40 min
python scripts/ab_eval_v3.py          # A/B eval

See full scripts in the original training repo (will be linked if public).

Acknowledgments

• Base model: Qwen2.5-7B-Instruct (4-bit quantized by mlx-community)
• Training data: HKAllen/cantonese-chinese-parallel-corpus, raptorkwok/cantonese-chinese-parallel-corpus
• Hardware: M1 Max 32GB with Apple MLX framework