Downloads · 30 days
0
ThakiCloud/Qwen3.8-27B-ko-cjk-suppressed
Qwen3.8-27B-ko-cjk-suppressed is a machine learning model from ThakiCloud. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Reduces, at the weight level, the phenomenon where Hanja/kana leak into the middle of a sentence when answering in Korean. Only 54,902 CJK-token rows of lmhead are replaced.
Downloads · 30 days
0
Access
Public
Updated Sep 10, 2026
Repo size
—
Likes
2
Public
Click a slice to open those files.
.json1.2 MB · 97%
From the Hugging Face model README
Reduces, at the weight level, the phenomenon where Hanja/kana leak into the middle of a
sentence when answering in Korean. Only 54,902 CJK-token rows of lm_head are replaced.
인기가极高的 해양 생물 테마파크 ← base (a very popular marine-life theme park)
오후 6시前后的(경계) 시간대는 ← base (the time slot around 6pm)
파일 버전号和 마지막 수정 시각을 ← base (the file version number and last-modified time)
Only one tensor, lm_head.weight, changes, and 31 of the 32 files (55.6GB total) are
byte-identical to the original. Rather than making another 55GB copy, it's better to download
the original and run the script here — it follows the base whenever it's updated, and diff
shows exactly what changed.
hf download Qwen/Qwen3.8-27B --local-dir ./Qwen3.8-27B
python apply_mask.py --model-dir ./Qwen3.8-27B --mask mask_m2-medium.json
Takes a few minutes on CPU (no GPU needed). The script verifies the margin before writing,
and after writing it reloads and asserts that no unintended tensor changed. --dry-run lets you
measure only, without writing.
| File | What |
|---|---|
apply_mask.py | Script applied to the original (measure → verify → patch 1 shard → verify on reload) |
mask_m2-medium.json | Adopted mask — 54,902 tokens |
mask_m1-conservative.json | Conservative mask — 34,187 tokens (kana + simplified-only) |
mask_m3-full.json | Full mask — 58,888 tokens (⛔ breaks Hanja glosses, see below) |
build_masks.py | Script that rebuilds the 3 masks from the tokenizer |
Measured paired, 3,369 prompts, same engine (temperature=0.0, max_tokens=400).
| Contamination rate | True errors* | |
|---|---|---|
Qwen/Qwen3.8-27B | 2.55% | 1.81% |
| M1 applied | 1.51% | 0.83% |
| M2 applied (recommended) | 0.68% | 0.18% |
| M3 applied | 0.06% | — |
* True errors excluding legitimate Korean Hanja glosses (개항(開港)). McNemar p < 0.0001,
5.7x the run-to-run noise floor of 0.33pp.
Korean Hanja glosses are preserved. On 12 prompts that explicitly requested Hanja, M1 and
M2 generate 12/12 identically to base. CJK input is also read unchanged — embed_tokens
and the tokenizer are not touched.
coding (HumanEval) base 92.19% = M2 92.19%, delta 0.0pp (n=64).
⚠️ At n=64 the minimum detectable difference is 13.29pp, so this is formally
INSUFFICIENT_N — read it as "not detected," not as "no regression."
Replaces the target rows of lm_head.weight with a large negative multiple of the mean
hidden-state direction.
W_i := -alpha * mu_h / ||mu_h||^2 (alpha = 200)
lm_head has no bias, so zeroing a row only makes its logit 0, not −inf. The moment every
other candidate is negative, 0 becomes the argmax. mu_h is measured as the mean final
hidden state over 6 Korean sentences (PROBE_TEXTS in apply_mask.py; across two independent
measurements ||mu_h||² reproduced within 0.4%, at 9845.958 / 9887.059). The resulting logit is
−217.0 versus a max of 16.5.
⛔ Using "the mean W row of Hangul tokens" as a proxy for mu_h was tried and rejected
(cos(mu_h, mu_W) = 0.048, essentially orthogonal). A forward pass is required.
By character, 72.3% of contamination is traditional/shared Hanja that Korean also uses, so
"cut simplified-only" alone isn't enough. But by token, 355 of 498 contamination events are
whole Chinese-word tokens like 您的·贵公司·具体时间. Korean Hanja glosses are split
into individual characters (개항(開港) → 開, 港).
So M2 cuts kana + simplified-only + pure-Hanja tokens of 2+ characters (54,902 of them) and
keeps single-Hanja tokens. The cost is Korean Sino-Korean words that happen to be single tokens,
like 목적·사고.
M3 (full, 58,888 tokens) brings contamination down to 0.06%, but Hanja glosses collapse from
12/12 to 3/12, and even the remaining 3 produce wrong Hanja (債權→倖, 새옹지마→壺齋).
It's included for comparison only and is not recommended.
Token ids are tied to the tokenizer. For a different checkpoint, rebuild the mask.
python build_masks.py --model <your/model> # requires transformers, opencc
temperature. Temperature drives the contamination rate by 5x — on the
same model, same prompts, T=1.0 gives 9.33% vs T=0.0 gives 1.92%. All numbers in this card
are at T=0.0.鄕·蕩. Rare Hanja have no standalone token in the vocabulary and are
assembled from byte fragments (鄕 → [98248, 243]), and those byte tokens cannot be cut
— cutting them breaks arbitrary UTF-8 handling.longctx·kmmlu·english
were not measured.・ in code context, not Korean context. The
27B model shows zero contamination on the same code-context probe, but don't assume this
holds unchanged on small models. Why only 0.8B breaks through is still unknown.⛔ The approach itself is not novel.
dnotitia/smoothie-qwen already distributes
pre-adjusted Qwen3 0.6B–235B checkpoints in the same direction (adjusting Qwen's lm_head to
suppress Chinese).
What this repo adds is the M1/M2/M3 mask curve, designed around Korean Hanja-gloss preservation as an explicit design axis, and the three observations noted above (temperature drives it 5x · the floor isn't 0 because of byte assembly · the same recipe doesn't reproduce on 0.8B).
Related: SASFT (ICLR 2026) · Korean token pruning · TLPO (ACL 2026)
Repos that redesign the same technique per target language. On 2026-09-01 the remaining five languages were also measured (300 prompts each, paired T=0/1.0 measurement).
| Language | Repo | Criterion | Measurement |
|---|---|---|---|
| Korean | (this repo) | simplified/kana + pure-Hanja 2+ chars | ✅ 3,369 prompts |
| Japanese | ja | Shift-JIS repertoire | △ 5.0→0.33% (borderline) |
| Traditional Chinese | zh-TW | Big5 repertoire | ✅ 40.0→1.33% (T=1.0) |
| Cantonese | yue | Big5-HKSCS repertoire | ✅ 20.33→0.67% (T=1.0) |
| Vietnamese | vi | full CJK (zero legitimate use) | △ 2.0→0.0% (borderline) |
| Arabic | ar | iso8859-6 + tashkil/digit exceptions | ⛔ contamination not observed (null) |
⛔ Do not reuse this repo's mask on another language as-is. 11 of 12 common Japanese words
get cut by this M2 mask — the rule "pure-Hanja token of 2+ characters = Chinese word" only holds
under Korean orthography. build_masks_multi.py holds the per-language rules.
apache-2.0 — same as the base model Qwen/Qwen3.8-27B. This repo distributes only the mask
and scripts, and does not redistribute weights.