Qwen3.5-9B-GLM-Wannabe

Experimental Qwen3.5-9B Full Finetune with GLM Data

An experimental full finetune of Qwen3.5-9B using distilled data from GLM 4.7 and GLM 5, built to test my pipeline for linear attention training, specifically the gated delta mechanism. It has no benchmark goals and has not been evaluated on any downstream tasks.

---

Training Data

A mix of three sources:

• Jackrong/glm-4.7-multiturn-CoT: Multi-turn CoT distillation from GLM-4.7. Each conversation was split by turn, and thinking traces were stripped from all intermediate assistant turns. Only the final assistant turn retains the full <think>...</think> trace.
• TeichAI/Pony-Alpha-15k: GLM-5 distilled single-turn data, filtered to retain only examples where the thinking trace does not exceed 12K tokens.
• Self-generated: Simple question-answer pairs so the model doesn't overthink.

For the non-reasoning mode, the same data was reused with empty thinking traces (<think>\n\n</think>) to match the Qwen3.5 chat template.

Preprocessing: Data was reformatted from Gemini/GLM markdown style (* asterisk bullet points → -, indentation spacing adjusted) to match Qwen conventions for faster convergence.

Context Length

Max context length is inherited from Qwen3.5-9B, which natively supports up to 262,144 tokens. However, since training was conducted at 16K sequence length, quality may degrade at longer contexts.

---

Training Setup

Implementation Notes

• Gated delta (linear attention): Compiled with torch_xla.experimental.scan
• Standard attention: Accelerated with torch_xla.experimental.custom_kernel.flash_attention