Downloads · 30 days
17
4% of all-time downloads
macadeliccc/Nous-Hermes-2-Mixtral-8x7B-DPO-HQQ
Nous-Hermes-2-Mixtral-8x7B-DPO-HQQ is a text generation model from macadeliccc. Use it when you need the model to write or continue text. It is set up for transformers.
This model is part of a series of HQQ tests. I make no claims on the performance of this model and it very well may change or be deleted.
Downloads · 30 days
17
4% of all-time downloads
All-time downloads
449
Public
Parameters
263M
527 MB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors527 MB · 100%
From the Hugging Face model README
This model is part of a series of HQQ tests. I make no claims on the performance of this model and it very well may change or be deleted.
This is a very extreme example of quantization.
from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('macadeliccc/Nous-Hermes-2-Mixtral-8x7B-DPO-HQQ', trust_remote_code=True)
model = HQQModelForCausalLM.from_pretrained(
"macadeliccc/Nous-Hermes-2-Mixtral-8x7B-DPO-HQQ",
torch_dtype=torch.float16,
device_map="auto"
)