Downloads · 30 days
35
11% of all-time downloads
hybrid-moe-30b-a3b-base
geodesic-research/hybrid-moe-30b-a3b-base
hybrid-moe-30b-a3b-base is a text generation model from geodesic-research. Use it when you need the model to write or continue text.
A base language model with a hybrid architecture of Mamba2, attention and mixture-of-experts layers, trained from random initialisation.
All-time downloads
319
Public
Parameters
31.6B
63.2 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors63.2 GB · 100%
Parameter types
How the weights are stored.
BF1631.6B · 100%
Try a prompt
Model card
From the Hugging Face model README
hybrid-moe-30b-a3b-base
A base language model with a hybrid architecture of Mamba2, attention and mixture-of-experts layers, trained from random initialisation.
| Architecture | Hybrid Mamba2 + attention + mixture-of-experts (model_type: nemotron_h) |
| Layers | 52 |
| Hidden size | 2,688 |
| Parameters | 31.6B total, roughly 3B active per token |
| Initialisation | Random (trained from scratch) |
| Tokens seen | 553,765,568,512 (553.8B) |
| Context length | Trained at 8,192 tokens, then continued at 32,768 |
| Precision | BF16 |
This is a base model: it has had no instruction tuning or preference optimisation, so it is suited to text completion rather than chat.
- Task
- Text Generation
- Type
- nemotron_h
- Languages
- en
- Created
- Sep 16, 2026
- Updated
- Sep 16, 2026