Downloads · 30 days
6
8% of all-time downloads
Demuxtral-1-22B-v0.1
lodrick-the-lafted/Demuxtral-1-22B-v0.1
Demuxtral-1-22B-v0.1 is a text generation model from lodrick-the-lafted. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Mixtral-8x22B with a bit of tensor surgery/lobotomy to chop out all but one of the MoE FFNs and configured to act like a dense model. Is it useful? Not in this state, without further training of the FFN or merging a f…
All-time downloads
78
Public
Parameters
22.2B
44.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors44.5 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- mistral
- Access
- Public
- Created
- Apr 14, 2024
- Updated
- Apr 15, 2024
- SHA
- a756f0ee
Try a prompt
Model card
From the Hugging Face model README
Mixtral-8x22B with a bit of tensor surgery/lobotomy to chop out all but one of the MoE FFNs and configured to act like a dense model. Is it useful? Not in this state, without further training of the FFN or merging a few models together. When testing it, the only tokens it wanted to output was space and carriage return. Enjoy. <br/>
There are 8 variations as you might expect. <br/>
[https://hugggingface.co/lodrick-the-lafted/Demuxtral-1-22B-v0.1]<br/> [https://hugggingface.co/lodrick-the-lafted/Demuxtral-2-22B-v0.1]<br/> [https://hugggingface.co/lodrick-the-lafted/Demuxtral-3-22B-v0.1]<br/> [https://hugggingface.co/lodrick-the-lafted/Demuxtral-4-22B-v0.1]<br/> [https://hugggingface.co/lodrick-the-lafted/Demuxtral-5-22B-v0.1]<br/> [https://hugggingface.co/lodrick-the-lafted/Demuxtral-6-22B-v0.1]<br/> [https://hugggingface.co/lodrick-the-lafted/Demuxtral-7-22B-v0.1]<br/> [https://hugggingface.co/lodrick-the-lafted/Demuxtral-8-22B-v0.1]<br/>
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- License
- apache-2.0
- Created
- Apr 14, 2024
- Updated
- Apr 15, 2024