Qwen3.5-0.8B-vocabulary-trimming

Qwen3.5-0.8B-Trimmed

A vocabulary-trimmed version of Qwen3.5-0.8B with 41.3% fewer tokens, keeping only Latin (EN, FR, DE, ES, IT, PT, NL, PL, RO, HU) and Greek scripts while preserving all original capabilities in those languages.

---

📊 Trimming Statistics

Original vocabulary distribution by script

---

🎯 Why Trim?

Eliminating random Asian characters

One of the most common issues with general multilingual models is the sporadic appearance of Chinese, Japanese, or Korean characters in outputs meant for Latin-script languages — even when the prompt has nothing to do with those languages. This happens because the model occasionally explores parts of the vocabulary that are never actively used for the target task. By permanently removing those tokens, this issue is resolved at a structural level: the model simply cannot generate them anymore, as they no longer exist in its output space.

Faster generation and prompt processing

Reducing the vocabulary size has a direct and measurable impact on speed:

• Prompt processing (prefill) — The embedding matrix is smaller, which reduces load time and memory access during the prefill phase.
• Token-by-token generation (decoding) — The search for the best token in the output distribution (softmax + argmax) operates over a 41.3% smaller space. This translates into faster decoding, especially on hardware where the softmax computation is the bottleneck.

Reduced memory footprint

• Storage — The model file is 400.4 MB smaller (10.33% reduction).
• RAM / VRAM — The embedding matrix (lm_head + embed_tokens) occupies proportionally less memory. On systems with tight memory constraints, this reduction can make it possible to run the model where it previously wouldn't fit, or free up memory for other processes.

Potential performance improvement (1–3%)

Trimming the vocabulary may lead to a slight improvement in overall model performance on the retained languages. The hypothesis is that reducing the output space concentrates probability mass more effectively on relevant tokens, reducing the statistical noise introduced by thousands of tokens that are never used in the task context. Gains in the range of 1–3% may be observed on Latin-language benchmarks (perplexity, answer accuracy, coherence).

---

🔧 Methodology

Trimming was performed using the [vocab-trimmer] (release soon) tool, which follows this pipeline:

1. Vocabulary analysis — Each token in the original tokenizer is classified by Unicode script (Latin, CJK, Cyrillic, Arabic, etc.).
2. Selective filtering — Only Latin and Greek scripts are kept. Special tokens (EOS, BOS, PAD, etc.) and multimodal tokens are always preserved.
3. ID remapping — A continuous old_id → new_id mapping is built to compact the vocabulary with no gaps.
4. Weight slicing — Only the rows of the embedding matrices corresponding to retained tokens are extracted. The lm_head and embed_tokens layers are recalibrated.
5. Configuration reconstruction — tokenizer_config.json, tokenizer.json, config.json, and the vocabulary are rebuilt with the new IDs. Chat templates and special tokens are preserved verbatim.
6. Verification — Consistency between tokenizer and weights is validated (special token IDs < vocab_size, functional chat template, etc.).

---

⚠️ Limitations

• Unsupported languages — This model cannot generate text in Chinese, Japanese, Korean, Russian, Arabic, Thai, Hindi, Bengali, Hebrew, or any other language whose tokens were removed. Prompts in these languages will not be understood correctly.
• Supported languages — Functional languages include English, French, German, Spanish, Italian, Portuguese, Dutch, Polish, Romanian, Hungarian, and Greek.
• Code & Math — Programming and mathematical tokens are preserved (they primarily use Latin encoding). No loss of capability is expected in these domains.
• Emojis & Symbols — Common Unicode symbols and emojis are retained if they fall within the selected scripts.

---

💡 Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "your-namespace/Qwen3.5-0.8B-Trimmed"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

inputs = tokenizer("Explain quantum computing in simple terms.", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

📝 License

This model is derived from Qwen/Qwen3.5-0.8B and follows its original license. The trimming tool [vocab-trimmer]release soon is available under the MIT License.

---

🔗 References

• Original model : Qwen/Qwen3.5-0.8B
• Trimming tool : [vocab-trimmer] soon
• Qwen Team : Alibaba Cloud – Qwen Team