Qwen3.5-0.8B-PureText-ZH-EN-v5.2

Qwen3.5-0.8B Pure Text ZH-EN

A text-only derivative of Qwen/Qwen3.5-0.8B optimized for Chinese and English inference.

Changes

• Converted Qwen3_5ForConditionalGeneration to Qwen3_5ForCausalLM.
• Removed the vision tower and MTP weights.
• Removed visual, grounding, audio, and TTS special tokens.
• Pruned the vocabulary from 248,320 rows to 182,684 real tokens, padded to 182,784 rows for alignment.
• Remapped tokenizer IDs and retained old2new.json for converting existing tokenized datasets.
• Preserved the original weights exactly for every retained vocabulary item; no additional training was performed.

The resulting checkpoint has 685,284,160 parameters and occupies approximately 1.3 GiB in BF16 safetensors format.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "wejyy888/Qwen3.5-0.8B-PureText-ZH-EN-v5.2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")

messages = [{"role": "user", "content": "介绍一下杭州"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use a Transformers version that includes Qwen3_5ForCausalLM support.

Compatibility notes

• Verified environments: transformers 5.2.0 (tokenizers 0.22.2, torch 2.14.0) and 5.17.0. Under 5.2.0 the tokenizer loads at exactly 182,684 tokens with no automatic vocabulary expansion, all 320 tensors load as Qwen3_5ForCausalLM (685,284,160 parameters), and forward/generation smoke tests pass.

• The linear-attention layers fall back to a pure-torch path when causal-conv1d and flash-linear-attention are not installed; install both for faster training and inference.

• Previously tokenized datasets must be remapped with old2new.json.

• Chinese and English tokenization is preserved exactly after ID remapping.

• Other languages still decode correctly through byte fallback, but generally use more tokens.

• This checkpoint accepts text only; image, video, audio, and TTS inputs are unsupported.

• A short continued-pretraining or SFT stage is recommended before production use to adapt to the reduced vocabulary.

Evaluation

Chinese e-commerce live-search query intent classification: 100 queries, 3 classes (LIVE / BRAND / ITEM), greedy decoding, max 256 new tokens, evaluated with transformers 5.2.0.

Overall accuracy 81.0%, macro-F1 79.8%, and 97/100 per-row agreement with the unpruned source model under the identical prompt. Vocabulary pruning caused no measurable degradation on this task.

Reproduction

prune_qwen35_zh_en.py contains the conversion procedure. The padded vocabulary rows are initialized to zero so they cannot produce invalid high logits.

Validation

• Loaded successfully through AutoModelForCausalLM as Qwen3_5ForCausalLM.
• No visual or MTP parameters remain.
• Retained-token logits match the source model exactly in the comparison test.
• Chinese/English tokenizer equivalence, text chat-template encoding, forward pass, and generation smoke tests pass.
• 100-query intent classification end-to-end run under transformers 5.2.0 on Apple MPS reproduces the source model's accuracy (81.0%) with 97% per-row agreement.