Downloads · 30 days
9
3% of all-time downloads
mobiuslabsgmbh/Hermes-3-Llama-3.1-70B_4bitgs64_hqq
Hermes-3-Llama-3.1-70B_4bitgs64_hqq is a text generation model from mobiuslabsgmbh. Use it when you need the model to write or continue text. The card lists the license as llama3.1.
This is an <a href="https://github.com/mobiusml/hqq/"HQQ</a all 4-bit (group-size=64) quantized <a href="https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-70B" Hermes-3-Llama-3.1-70B</a model.
Downloads · 30 days
9
3% of all-time downloads
All-time downloads
359
Public
Repo size
42.7 GB
Likes
4
Public
Click a slice to open those files.
.pt42.7 GB · 100%
From the Hugging Face model README
This is an <a href="https://github.com/mobiusml/hqq/">HQQ</a> all 4-bit (group-size=64) quantized <a href="https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-70B"> Hermes-3-Llama-3.1-70B</a> model.

Model size and decoding speed should be similar to the <a href="https://huggingface.co/mobiuslabsgmbh/Llama-3.1-70b-instruct_4bitgs64_hqq">Llama-3.1-70b-instruct_4bitgs64_hqq</a> version.
First, install the dependecies:
pip install git+https://github.com/mobiusml/hqq.git #master branch fix
pip install bitblas
Also, make sure you use at least torch 2.4.0 or the nightly build.
Then you can use the sample code below:
import torch
from transformers import AutoTokenizer
from hqq.models.hf.base import AutoHQQHFModel
from hqq.utils.patching import *
from hqq.core.quantize import *
from hqq.utils.generation_hf import HFGenerator
#Load the model
###################################################
model_id = 'mobiuslabsgmbh/Hermes-3-Llama-3.1-70B_4bitgs64_hqq'
compute_dtype = torch.bfloat16 #bfloat16 for torchao, float16 for bitblas
cache_dir = '.'
model = AutoHQQHFModel.from_quantized(model_id, cache_dir=cache_dir, compute_dtype=compute_dtype)
tokenizer = AutoTokenizer.from_pretrained(model_id, cache_dir=cache_dir)
quant_config = BaseQuantizeConfig(nbits=4, group_size=64, quant_scale=False, quant_zero=False, axis=1)
patch_linearlayers(model, patch_add_quant_config, quant_config)
#Use optimized inference kernels
###################################################
HQQLinear.set_backend(HQQBackend.PYTORCH)
#prepare_for_inference(model) #default backend
prepare_for_inference(model, backend="torchao_int4")
#prepare_for_inference(model, backend="bitblas") #takes a while to init...
#Generate
###################################################
#For longer context, make sure to allocate enough cache via the cache_size= parameter
gen = HFGenerator(model, tokenizer, max_new_tokens=1000, do_sample=True, compile="partial").warmup() #Warm-up takes a while
gen.generate("Write an essay about large language models", print_tokens=True)
gen.generate("Tell me a funny joke!", print_tokens=True)
gen.generate("How to make a yummy chocolate cake?", print_tokens=True)