Downloads · 30 days
30
9% of all-time downloads
LlaMoE-Medium
Crystalcareai/LlaMoE-Medium
LlaMoE-Medium is a text generation model from Crystalcareai. Use it when you need the model to write or continue text. It is set up for transformers.
<p align="center" <img src="https://huggingface.co/Crystalcareai/LlaMoE-Medium/resolve/main/resources/ddb-nye2T3C3vZwJJm1l6A.png" width="auto" title="LlaMoE-Medium model image" </p
All-time downloads
330
Public
Parameters
30.6B
61.2 GB on disk
Likes
3
Public
Repo makeup
Click a slice to open those files.
.safetensors61.2 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- deepseek
- Access
- Public
- Created
- Apr 29, 2024
- Updated
- Apr 29, 2024
- SHA
- adb187c2
Try a prompt
Model card
From the Hugging Face model README
<p align="center"> <img src="https://huggingface.co/Crystalcareai/LlaMoE-Medium/resolve/main/resources/ddb-nye2T3C3vZwJJm1l6A.png" width="auto" title="LlaMoE-Medium model image"> </p>
This is a 4x8b Llama Mixture of Experts (MoE) model. It was trained on OpenHermes Resort from the Dolphin-2.9 dataset.
The model is a combination of 4 Llama fine-tunes, using DeepSpeed-MoE's architecture. All experts are active for every token.
This is a VERY good model, somewhere in between 8B and Llama 70B in capability. Enjoy!
Thank you to:
CrusoeEnergy for sponsoring the compute for this project
My collaborators Eric Hartford and Fernando (has too many names) Neto
- Task
- Text Generation
- Library
- transformers
- Type
- deepseek
- Created
- Apr 29, 2024
- Updated
- Apr 29, 2024