Qwen3.5-4B-A3B-Multilingual-Distilled-v1

Qwen3.5-4B-A3B-Multilingual-Distilled-v1

Experimental multilingual layerwise-distilled checkpoint derived from sepsy070716/Qwen3.5-4B-A3B-Upcycled with Qwen/Qwen3.5-4B as the dense teacher.

Research checkpoint. All 32 MoE blocks received local dense-FFN distillation, but the complete language model has not yet received joint end-to-end recovery training. Local reconstruction gains are not equivalent to downstream benchmark gains.

Architecture

Multilingual distillation

Training used a balanced ODC-By-1.0 FineWeb/FineWeb2 sample in Korean, English, Mandarin Chinese, Japanese, Spanish, and German. Every layer received 24 local updates (four documents per language). Evaluation used a disjoint split with two documents per language and layer.

• Mean held-out dense-FFN relative-MSE improvement: 6.57%
• Layers improved: 32/32
• Every language improved on every layer: 192/192 comparisons

Language means: de 6.83%, en 5.12%, es 7.15%, ja 7.47%, ko 7.42%, zh 5.60%.

End-to-end language-model check

On four disjoint 128-token documents per language, mean causal-LM loss improved from 10.378 for the untrained upcycle to 9.162 for this checkpoint, an 11.72% relative reduction. All six languages improved. The dense teacher scored about 2.94, which quantifies the large remaining recovery gap.

A two-prompt greedy generation smoke test loaded and executed successfully, but the next four tokens were whitespace-dominated in both Korean and English. Therefore this checkpoint is not yet suitable for chat, completion, benchmark, or production claims. Joint end-to-end recovery is required.

Loading

from transformers import AutoTokenizer, Qwen3_5MoeForConditionalGeneration

model_id = "sepsy070716/Qwen3.5-4B-A3B-Multilingual-Distilled-v1"
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(model_id, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

See distillation_manifest.json and distillation_summary.json for complete per-layer and per-language measurements.