Downloads · 30 days
62
23% of all-time downloads
qwen3.5-0.8b-moe-from-scratch
kshitijthakkar/qwen3.5-0.8b-moe-from-scratch
qwen3.5-0.8b-moe-from-scratch is a machine learning model from kshitijthakkar. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Built from scratch following Sebastian Raschka's LLMs-from-Scratch style, then weights transferred from kshitijthakkar/qwen3.5-moe-0.87B-d0.8B.
All-time downloads
274
Public
Parameters
2.9B
8.1 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors8 GB · 100%
At a glance
- License
- apache-2.0
- Model type
- qwen3_5_moe
- Access
- Public
- Created
- Mar 4, 2026
- Updated
- Mar 4, 2026
- SHA
- 48759056
Base models
Model card
From the Hugging Face model README
qwen3.5-0.8b-moe-from-scratch
Built from scratch following Sebastian Raschka's LLMs-from-Scratch style, then weights transferred from kshitijthakkar/qwen3.5-moe-0.87B-d0.8B.
Architecture
| Property | Value |
|---|---|
| Base model | kshitijthakkar/qwen3.5-moe-0.87B-d0.8B |
| Layers | 24 |
| Hidden size | 2048 |
| Experts | 8 total, top-2 active |
| Total parameters | 2914 M |
| Attention | Hybrid (GatedDeltaNet + Gated Full Attention) |
Usage
from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained(
"kshitijthakkar/qwen3.5-0.8b-moe-from-scratch",
torch_dtype="bfloat16",
device_map="auto",
)
processor = AutoProcessor.from_pretrained("kshitijthakkar/qwen3.5-0.8b-moe-from-scratch")
messages = [{"role": "user", "content": "Explain the Qwen3.5 architecture."}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True,
tokenize=True, return_dict=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
How it was built
See the companion notebook qwen3_5_from_scratch.ipynb
for the full from-scratch implementation.
- Type
- qwen3_5_moe
- License
- apache-2.0
- Languages
- en
- Created
- Mar 4, 2026
- Updated
- Mar 4, 2026