Qwen3.6-8B-A1.6B

Qwen3.6-8B-A1.6B

This is a tiny version of Qwen/Qwen3.6-35B-A3B created for testing and development.

Model Details

• Base Model: Qwen/Qwen3.6-35B-A3B
• Architecture: qwen3_5_moe (Qwen3_5MoeForConditionalGeneration)
• Total Parameters: 7.81B
• Activated Parameters: ~1.57B (8 of 256 routed experts per token, plus shared expert)

This is a multimodal (vision-language) mixture-of-experts model with a hybrid linear-attention / full-attention text backbone. The tiny model preserves the full architecture of the original: hybrid attention pattern (linear + full), 256 routed experts with top-8 routing, a shared expert, and the vision tower.

Configuration Changes

Only depth was reduced; all widths (hidden size, expert count, MoE intermediate size, attention head dims, vocab) match the original to keep the architecture faithful.

The hybrid attention pattern (full_attention_interval=4) is preserved: text layers are [linear, linear, linear, full, linear, linear, linear, full], giving 6 linear-attention and 2 full-attention layers.

Checkpoint Structure

Sharded safetensors checkpoint with model.safetensors.index.json, matching the original repository layout. Routed experts are stored in the original fused format (one 3D tensor per layer):

• ...mlp.experts.gate_up_proj → [256, 1024, 2048]
• ...mlp.experts.down_proj → [256, 2048, 512]

All non-MTP tensor name patterns match the original checkpoint exactly. The MTP (multi-token-prediction) layers from the original are intentionally omitted.

Usage

import torch
from transformers import Qwen3_5MoeForConditionalGeneration, AutoTokenizer

model = Qwen3_5MoeForConditionalGeneration.from_pretrained(
    "inference-optimization/Qwen3.6-8B-A1.6B", dtype=torch.bfloat16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3.6-8B-A1.6B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

1. Reduced text depth to 8 layers (preserving the hybrid attention pattern) and vision depth to 2, keeping all other dimensions.
2. Randomly initialized weights, then fine-tuned on a small toy text dataset until perplexity converged (train loss ≈ 0.02, perplexity ≈ 1.0).
3. Converted the fine-tuned checkpoint's per-expert tensors into the original fused-expert format and re-sharded to match the original repo layout.

Validation

• Loads with Qwen3_5MoeForConditionalGeneration (no missing/unexpected keys).
• Perplexity on the validation text ≈ 1.0 (target ≤ 10).
• Greedy generation is coherent on the fine-tuning distribution.

Notes

• The weights are randomly initialized and fine-tuned only on a tiny toy dataset; this model is for testing and development only and has no real-world language or vision capability.
• The vision tower is included for architecture coverage but was not trained on image data.