CantoneseLLM-v2.0-8B-Thinking

CantoneseLLM-v2.0-8B-Thinking

An 8B dense model based on Qwen3-8B that reasons and answers in Hong Kong Cantonese.

It is the final checkpoint of a five-stage pipeline: continual pre-training → chat-vector merge → supervised fine-tuning → DPO → two-stage RLVR. The full development history, including the stages that failed and why, is documented in the technical report.

📄 Paper: CantoneseLLM v2: Reasoning in a Low-Resource Language (arXiv:2609.06970)

🧪 Evaluation Benchmark: hon9kon9ize/hkeval2025

Reasoning in Cantonese

Open models prompted in Cantonese answer in Cantonese but reason in another language. Here is the same translated GSM8K probe, under a system prompt that explicitly asks for Hong Kong Cantonese.

Chat-vector merged checkpoint — reasons entirely in Simplified Chinese, despite the system prompt and despite continual pre-training on Traditional Chinese and Cantonese:

嗯，用户问的是Janet每天赚多少钱。首先，我需要理清楚整个流程。题目说Janet的鸭子每天生16只蛋。

(Well, what the user is asking is how much Janet earns each day. First, I need to work through the whole process. The problem says Janet's ducks lay 16 eggs a day.)

This model — reasons in Cantonese end to end, with the Cantonese classifier 隻 zek3 in place of 只 and colloquial forms such as 剩返 zing6 faan1:

呢個係一個數學應用題，我需要先理解題目入面嘅數據同埋計算邏輯。Janet 隻鴨每日生 16 隻蛋。首先，佢自己食早餐用咗 3 隻，然後用 4 隻嚟整鬆餅。… 剩返嘅蛋就係 16 − 7 = 9 隻。

(This is a mathematics word problem, and I need to understand the figures and the logic of the calculation first. Janet's ducks lay 16 eggs a day. First, she uses 3 for her own breakfast, then 4 to make muffins. … The eggs left over are 16 − 7 = 9.)

Both reach the correct answer, 18 dollars. What changes is where the reasoning happens. The 只/隻 substitution is not cosmetic: 只 can only be read zi2 in Hong Kong Cantonese, which is a different word from the classifier the sentence needs.

The mechanism is a multiplicative language and script term in the RLVR reward — task_reward × language_multiplier × format_factor — so a wrong-language trace cannot buy its way back by being correct.

Trace length over eight fixed probes (mean tokens between the reasoning tags):

The SFT row is not a typo. At 8B, supervised fine-tuning on a mixture dominated by another language's reasoning tokens made the model emit an empty reasoning block on 64.5% of generations — the empty <think></think> pair is a near-deterministic two-token continuation and became a low-loss attractor. DPO restored the block; RLVR restored the language.

---

Benchmark results

HKCanto-Eval (Cheng et al., 2025), reasoning mode on:

Read this table carefully. SFT cost 20.52 points. DPO recovered 10.19 and RLVR a further 3.62, leaving the model 6.71 points below the merged checkpoint it started from. The 30B-A3B closed the same gap to 1.20 points; this model did not. What it gained instead is Cantonese reasoning, which this benchmark does not measure.

The paper's stated reason is that teaching a model to reason in a new language needs Cantonese reasoning traces present during continual pre-training, not only in post-training — and the 8B had the least headroom to absorb that gap.

If you can run the 30B-A3B, run that one instead. This model is released for deployments where 8B dense is the constraint, and because the contrast between the two sizes is one of the paper's findings.

---

Artefacts released with this work

Every post-training stage on this card is reproducible from public components. The CPT corpus itself is not released, but its two largest public constituents are.

The three-way parallel structure of the RL corpus is what makes the cross-language pass-rate gap reported in the paper measurable: for the translated environments, prompt language is the only variable.

Other models in this release

All four checkpoints are in the CantoneseLLM v2.0 collection.

---

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

SYSTEM = "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "小明有 5 個蘋果，佢俾咗 2 個朋友，每人 1 個，跟住又買多 3 個。佢而家有幾多個蘋果？"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95)
print(tokenizer.decode(out[0][len(inputs.input_ids[0]):], skip_special_tokens=True))

vLLM

The stock command works — no special flags are required:

vllm serve hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking

Add --tensor-parallel-size N to shard across N GPUs, and --reasoning-parser qwen3 if you want the reasoning block returned separately as reasoning_content rather than inline in content (parser names vary by vLLM version).

OpenAI-compatible API

The served endpoint speaks the OpenAI protocol, so the official client works unchanged:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

SYSTEM = "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"

resp = client.chat.completions.create(
    model="hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking",
    messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": "小明有 5 個蘋果，佢俾咗 2 個朋友，每人 1 個，跟住又買多 3 個。佢而家有幾多個蘋果？"},
    ],
    temperature=0.6,
    top_p=0.95,
    max_tokens=2048,
)

msg = resp.choices[0].message
print(getattr(msg, "reasoning_content", None) or "")  # populated with --reasoning-parser
print(msg.content)

Or with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hon9kon9ize/CantoneseLLM-v2.0-8B-Thinking",
    "messages": [
      {"role": "system", "content": "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"},
      {"role": "user", "content": "點解香港嘅雨季集中喺五月到九月？"}
    ],
    "temperature": 0.6, "top_p": 0.95, "max_tokens": 2048
  }'

Sampling: temperature 0.6, top-p 0.95 — the settings used for every evaluation reported above and in the paper. Avoid greedy decoding.

System prompt: the Cantonese system prompt above was used throughout training and evaluation. Behaviour with other system prompts, or with none, is not characterised.

Thinking and non-thinking. Unlike the 30B-A3B, 25% of this model's SFT data carried no chain-of-thought, deliberately, so that it retains the ability to answer without an explicit reasoning block. It is the closer of the two to stock Qwen3 hybrid behaviour — but given the empty-block history above, verify the mode switch on your own prompts rather than assuming it.

---

Training pipeline

Released weights are the final step of stage 2, not a best-validation checkpoint.

---

Risks & Limitations

• Does not fully recover pre-SFT benchmark performance — 6.71 points below the merged checkpoint. See the table above. The 30B-A3B closed this gap; this model did not.
• Short reasoning traces. 232 tokens on average, at a CoT-to-answer ratio of 1.09. No long original Cantonese reasoning traces existed at the scale SFT needed; human-written or human-verified traces amounted to 131 rows. Most Cantonese CoT in training was machine-translated from English or Simplified Chinese.
• History of empty reasoning blocks. The SFT checkpoint emitted an empty block on 64.5% of generations. DPO and RLVR repaired this, but if you fine-tune further from here, watch for the attractor returning.
• No Hong Kong grounding in the reasoning data. The SFT mixture contained no content grounded in Hong Kong entities or current events — that knowledge comes from CPT only.
• Benchmark scores are underestimates of knowledge. DPO's pair-selection weighted longer responses with no reward for instruction compliance, so the model sometimes ignores "answer with the letter only". Multiple-choice parsers that read a bare leading option letter will under-score it. Use a marker-anchored extractor.
• One run per stage. Severe compute constraints meant no hyperparameter sweep in any post-training stage.
• Preference data were model-judged, not human-annotated, and are bounded by the judge's own Cantonese ability.
• Long context is inherited and untested. Training sequence lengths were 8,192 (SFT/DPO) and 16,384 (RLVR stage 2). Behaviour beyond that is whatever the Qwen3 base provides.
• Standard LLM caveats apply: it will hallucinate, and it has not been safety- tuned beyond what the chat vector carried over.

---

Citation

@misc{cantonesellm_v2,
      title={CantoneseLLM v2: Reasoning in a Low-Resource Language},
      author={Tsz Chung Cheng and Chung Shing Cheng and Chaak Ming Lau and Cheuk Hei Chong},
      year={2026},
      eprint={2609.06970},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.06970},
}

Acknowledgements

Continual pre-training (CPT) was carried out on Cloud TPUs (Tensor Processing Units) from Google's TPU Research Cloud with MaxText. Post-training was carried out on computer resources offered under the category of General Projects by Research Institute for Information Technology, Kyushu University. Usage fee and cost of data-curation costs with proprietary APIs were covered by Votee AI

The RLVR stage builds on NVIDIA's NeMo-RL and NeMo-Gym, and on the Nemotron post-training and RL datasets.