Downloads · 30 days
283
2% of all-time downloads
quad-mixtrals-gguf
nisten/quad-mixtrals-gguf
quad-mixtrals-gguf is a machine learning model from nisten. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Experimental quants of 4 expert MoE mixtrals in various GGUF formats.
All-time downloads
17.8K
Public
Repo size
245 GB
Likes
31
Public
Repo makeup
Click a slice to open those files.
.gguf245 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Dec 22, 2023
- Updated
- Dec 27, 2023
- SHA
- 678c9e97
Model card
From the Hugging Face model README
Experimental quants of 4 expert MoE mixtrals in various GGUF formats.
Original model used for custom quants: NeverSleep/Mistral-11B-SynthIAirOmniMix
https://huggingface.co/NeverSleep/Mistral-11B-SynthIAirOmniMix
Goal is to have the best performing MoE < 10gb
Experimental q8 and q4 files for training/finetuning too.
No sparsity tricks yet.
8.4gb custom 2bit quant works ok up until 512 token length then starts looping.
- Install llama.cpp from github and run it:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j
wget https://huggingface.co/nisten/quad-mixtrals-gguf/resolve/main/4mixq2.gguf
./server -m 4mixq2.gguf --host "my.internal.ip.or.my.cloud.host.name.goes.here.com" -c 512
limit output to 500 tokens
- License
- apache-2.0
- Created
- Dec 22, 2023
- Updated
- Dec 27, 2023