Downloads · 30 days
8
1% of all-time downloads
mobiuslabsgmbh/Llama-2-70b-hf-2bit_g16_s128-HQQ
Llama-2-70b-hf-2bit_g16_s128-HQQ is a text generation model from mobiuslabsgmbh. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
This is a version of the LLama-2-70B-hf model quantized to 2-bit via Half-Quadratic Quantization (HQQ): https://mobiusml.github.io/hqqblog/
Downloads · 30 days
8
1% of all-time downloads
All-time downloads
566
Public
Repo size
26.9 GB
Likes
2
Public
Click a slice to open those files.
.pt26.9 GB · 100%
From the Hugging Face model README
This is a version of the LLama-2-70B-hf model quantized to 2-bit via Half-Quadratic Quantization (HQQ): https://mobiusml.github.io/hqq_blog/
This model outperforms an fp16 LLama-2-13B (perplexity 4.13 vs. 4.63) for a comparable ~26GB size.
Warning: this model is deprecated, it requires an older version of hqq.
To run the model, install the HQQ library:
#This model is deprecated and requires older versions
pip install hqq==0.1.8
pip install transformers==4.46.0
and use it as follows:
model_id = 'mobiuslabsgmbh/Llama-2-70b-hf-2bit_g16_s128-HQQ'
from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = HQQModelForCausalLM.from_quantized(model_id)
Limitations: <br> -Only supports single GPU runtime.<br> -Not compatible with HuggingFace's PEFT.<br>