Downloads · 30 days
29
32% of all-time downloads
AlexWortega/moe-200m-qwen3-step365000-12B-20260525-2225
moe-200m-qwen3-step365000-12B-20260525-2225 is a text generation model from AlexWortega. Use it when you need the model to write or continue text. It is set up for pytorch. The card lists the license as apache-2.0.
Sparse-MoE LM with ~200M active / ~616M total parameters, trained from-scratch on Ultra-FineWeb (English split), tokenizer Qwen/Qwen3-0.6B-Base (vocab 151,936).
Downloads · 30 days
29
32% of all-time downloads
All-time downloads
91
Public
Parameters
616M
2.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2.5 GB · 100%
From the Hugging Face model README
Sparse-MoE LM with ~200M active / ~616M total parameters,
trained from-scratch on Ultra-FineWeb (English split), tokenizer
Qwen/Qwen3-0.6B-Base
(vocab 151,936).
model.py and config.json. 1 shared + 16
routed experts top-2, GQA(10/2)×16 layers, partial RoPE, QK-Norm,
SwiGLU, tied embeddings.| key | value |
|---|---|
| param_count_total | 616,471,632 |
| param_count_active | ~203,591,000 (sum(p.numel()) minus inactive routed experts) |
| dataset | openbmb/Ultra-FineWeb (split=en, content) |
| tokenizer | Qwen/Qwen3-0.6B-Base (vocab=151,936) |
| step | 365000 |
| tokens_consumed | 11,960,320,000 |
EVAL_*.md).