Qwen3-Lite-3B-A0.9B

Qwen3-3B-A0.9B

This repository contains the current best local checkpoint exported from the Qwen3 MoE architecture exploration in this workspace.

Files

• Model weights in Hugging Face format
• Architecture config: qwen3_3p1b_a0p85b_moe_30biso_4l.json
• Recovery finetune config: recover_dialogue_qwen3_3p1b_30biso_recovery_cn_v1.yaml
• Smoke evaluation snapshot: candidate_v1_smoke_suite.json

Architecture Summary

• Base family: Qwen3 MoE
• Hidden size: 2048
• Layers: 4
• Attention heads: 32
• KV heads: 4
• Experts: 128
• Active experts per token: 8
• MoE intermediate size: 768
• Dense intermediate size: 6144
• Dtype: bfloat16

Current Best Local Status

This upload corresponds to the checkpoint currently documented as the best working local baseline in:

• README.md
• docs/stage1/qwen3_moe_4layer_recovery.md

Its practical status is:

• It can handle simple QA and part of short Chinese dialogue.
• It is not yet a fully repaired dialogue model.
• Later recovery branches did not consistently outperform this baseline.

Tokenizer

The tokenizer used during local experiments is the official Qwen tokenizer from the Qwen3-4B-Instruct release.

Tokenizer files are not re-exported in this checkpoint bundle because local training followed the same save style as the official weight export workflow.