Downloads · 30 days
19
4% of all-time downloads
mobiuslabsgmbh/Mixtral-8x7B-v0.1-hf-4bit_g64-HQQ
Mixtral-8x7B-v0.1-hf-4bit_g64-HQQ is a text generation model from mobiuslabsgmbh. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This is a version of the Mixtral-8x7B-v0.1 model (https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) quantized to 4-bit via Half-Quadratic Quantization (HQQ). To run the model, install the HQQ library:
Downloads · 30 days
19
4% of all-time downloads
All-time downloads
523
Public
Repo size
25.9 GB
Likes
1
Public
Click a slice to open those files.
.pt25.9 GB · 100%
From the Hugging Face model README
This is a version of the Mixtral-8x7B-v0.1 model (https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) quantized to 4-bit via Half-Quadratic Quantization (HQQ).
To run the model, install the HQQ library:
#This model is deprecated and requires older versions
pip install hqq==0.1.8
pip install transformers==4.46.0
and use it as follows:
model_id = 'mobiuslabsgmbh/Mixtral-8x7B-v0.1-hf-4bit_g64-HQQ'
#Load the model
from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = HQQModelForCausalLM.from_quantized(model_id)
#Optional
from hqq.core.quantize import *
HQQLinear.set_backend(HQQBackend.PYTORCH_COMPILE)