Downloads · 30 days
3
9% of all-time downloads
aldigobbler/smollmv2-360Mx4E-MoE-v0.1
smollmv2-360Mx4E-MoE-v0.1 is a machine learning model from aldigobbler. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Based off of smollmv2. (Llama) MoE-ified then further trained on a general dataset.
Downloads · 30 days
3
9% of all-time downloads
All-time downloads
34
Public
Repo size
2 GB
Likes
0
Public
Click a slice to open those files.
.bin2 GB · 100%
From the Hugging Face model README
Based off of smollmv2. (Llama) MoE-ified then further trained on a general dataset.
MoE layers: [8, 12, 16, 20, 24, 28]
Top-k: 2 (activates 50.0% of experts per token)
Hidden size: 960
Total parameters: 494,554,560
Trainable parameters: 494,554,560
Auxiliary loss weight: 0.01
code @ https://gist.github.com/cappuch/6a454ec8d2d349a27f9fd84f6ac90554