Downloads · 30 days
12
17% of all-time downloads
masao1211/karakuri-lm-70b-chat-v0.1-AWQ
karakuri-lm-70b-chat-v0.1-AWQ is a text generation model from masao1211. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
- Model creator: karakuri-ai - Original model: KARAKURI LM 70B Chat v0.1
Downloads · 30 days
12
17% of all-time downloads
All-time downloads
69
Public
Parameters
69.2B
37.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors37.1 GB · 100%
How the weights are stored.
I3268.5B · 99%
From the Hugging Face model README
This repo contains AWQ model files for KARAKURI LM 70B Chat v0.1.
I created AWQ model files by using used autoawq==0.2.3.
pip install autoawq==0.2.3
This is the Python code to create AWQ model.
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = "karakuri-ai/karakuri-lm-70b-chat-v0.1"
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
# Quantize
model.quantize(tokenizer, quant_config=quant_config, calib_data="mmnga/wikipedia-ja-20230720-1k")
quant_path = "karakuri-lm-70b-v0.1-AWQ"
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
from vllm import LLM, SamplingParams
sampling_params = SamplingParams(temperature=0.0, max_tokens=100)
llm = LLM(model="masao1211/karakuri-lm-70b-chat-v0.1-AWQ", max_model_len=4096)
system_prompt = "System prompt"
messages = [{"role": "system", "content": "System prompt"}]
messages.append({"role": "user", "content": "User Prompt"})
prompt = llm.llm_engine.tokenizer.tokenizer.apply_chat_template(conversation=messages, add_generation_prompt=True, tokenize=False)
prompts = [prompt]
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")