Downloads · 30 days
12
2% of all-time downloads
mobiuslabsgmbh/Llama-2-70b-chat-hf-2bit_g16_s128-HQQ
Llama-2-70b-chat-hf-2bit_g16_s128-HQQ is a text generation model from mobiuslabsgmbh. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
This is a version of the LLama-2-70B-chat-hf model quantized to 2-bit via Half-Quadratic Quantization (HQQ): https://mobiusml.github.io/hqqblog/
Downloads · 30 days
12
2% of all-time downloads
All-time downloads
667
Public
Repo size
26.9 GB
Likes
3
Public
Click a slice to open those files.
.pt26.9 GB · 100%
From the Hugging Face model README
This is a version of the LLama-2-70B-chat-hf model quantized to 2-bit via Half-Quadratic Quantization (HQQ): https://mobiusml.github.io/hqq_blog/
To run the model, install the HQQ library:
#This model is deprecated and requires older versions
pip install hqq==0.1.8
pip install transformers==4.46.0
and use it as follows:
model_id = 'mobiuslabsgmbh/Llama-2-70b-chat-hf-2bit_g16_s128-HQQ'
from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = HQQModelForCausalLM.from_quantized(model_id)
model_id = 'mobiuslabsgmbh/Llama-2-70b-chat-hf-2bit_g16_s128-HQQ'
from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = HQQModelForCausalLM.from_quantized(model_id)
##########################################################################################################
import transformers
from threading import Thread
from sys import stdout
def print_flush(data):
stdout.write("\r" + data)
stdout.flush()
#Adapted from https://huggingface.co/spaces/huggingface-projects/llama-2-7b-chat/blob/main/app.py
def process_conversation(chat):
system_prompt = chat['system_prompt']
chat_history = chat['chat_history']
message = chat['message']
conversation = []
if system_prompt:
conversation.append({"role": "system", "content": system_prompt})
for user, assistant in chat_history:
conversation.extend([{"role": "user", "content": user}, {"role": "assistant", "content": assistant}])
conversation.append({"role": "user", "content": message})
return tokenizer.apply_chat_template(conversation, return_tensors="pt").to('cuda')
def chat_processor(chat, max_new_tokens=100, do_sample=True):
tokenizer.use_default_system_prompt = False
streamer = transformers.TextIteratorStreamer(tokenizer, timeout=10.0, skip_prompt=True, skip_special_tokens=True)
generate_params = dict(
{"input_ids": process_conversation(chat)},
streamer=streamer,
max_new_tokens=max_new_tokens,
do_sample=do_sample,
top_p=0.90,
top_k=50,
temperature= 0.6,
num_beams=1,
repetition_penalty=1.2,
)
t = Thread(target=model.generate, kwargs=generate_params)
t.start()
outputs = []
for text in streamer:
outputs.append(text)
print_flush("".join(outputs))
return outputs
###################################################################################################
outputs = chat_processor({'system_prompt':"You are a helpful assistant.",
'chat_history':[],
'message':"How can I build a car?"
},
max_new_tokens=1000, do_sample=False)
<b>Output</b>:
<p> Building a car is a complex process that involves designing, prototyping, testing, and manufacturing. Here are some general steps you can follow to build a car:Please note that this is a high-level overview of the process, and building a car can be a complex and time-consuming task. It requires specialized knowledge, skills, and tools, as well as a clean and organized workspace. Additionally, safety precautions should always be taken when working on vehicles, as they can be dangerous if mishandled.
If you are not experienced in automotive construction, it may be advisable to seek guidance from professionals or take a course in automotive mechanics before attempting to build a car.
Limitations: <br> -Only supports single GPU runtime.<br> -Not compatible with HuggingFace's PEFT.<br>