Downloads · 30 days
5
3% of all-time downloads
skyxiaobaibai/Qwen1.5-7B-chat-gptq_model-4bit-128g
Qwen1.5-7B-chat-gptq_model-4bit-128g is a text generation model from skyxiaobaibai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
from transformers import AutoTokenizer, TextGenerationPipeline from autogptq import AutoGPTQForCausalLM, BaseQuantizeConfig import logging
Downloads · 30 days
5
3% of all-time downloads
All-time downloads
191
Public
Repo size
5.9 GB
Likes
0
Public
Click a slice to open those files.
.safetensors5.9 GB · 100%
From the Hugging Face model README
from transformers import AutoTokenizer, TextGenerationPipeline from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig import logging
logging.basicConfig( format="%(asctime)s %(levelname)s [%(name)s] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S" )
pretrained_model_dir = "Qwen/Qwen1.5-7B-Chat" quantized_model_dir = "/gptq_model-4bit-128g"
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True) examples = [ tokenizer( "auto-gptq is an easy-to-use model quantization library with user-friendly apis, based on GPTQ algorithm." ) ]
quantize_config = BaseQuantizeConfig( bits=4, # quantize model to 4-bit group_size=128, # it is recommended to set the value to 128 desc_act=False, # set to False can significantly speed up inference but the perplexity may slightly bad )
model = AutoGPTQForCausalLM.from_pretrained(pretrained_model_dir, quantize_config)
model.quantize(examples)
model.save_quantized(quantized_model_dir)
model.save_quantized(quantized_model_dir, use_safetensors=True)
model = AutoGPTQForCausalLM.from_quantized(quantized_model_dir, device="cuda:0")
print(tokenizer.decode(model.generate(**tokenizer("auto_gptq is", return_tensors="pt").to(model.device))[0]))
pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer) print(pipeline("auto-gptq is")[0]["generated_text"])