Downloads · 30 days
10
16% of all-time downloads
tctsung/TinyLlama-1.1B-chat-v1.0-awq
TinyLlama-1.1B-chat-v1.0-awq is a text generation model from tctsung. Use it when you need the model to write or continue text. It is set up for transformers.
This model is quantized by autoawq package using tctsung/chatrestaurantrecommendation as calibration dataset
Downloads · 30 days
10
16% of all-time downloads
All-time downloads
64
Public
Parameters
1.1B
766 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors766 MB · 100%
How the weights are stored.
I32969M · 88%
From the Hugging Face model README
This model is quantized by autoawq package using tctsung/chat_restaurant_recommendation as calibration dataset
Reference model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
For more details, see github repo tctsung/LLM_quantize
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
# load model & tokenizer:
model_id = "tctsung/TinyLlama-1.1B-chat-v1.0-awq"
model = LLM(model = model_id, dtype='half',
quantization='awq', gpu_memory_utilization=0.9)
sampling_params = SamplingParams(temperature=1.0,
max_tokens=1024,
min_p=0.5,
top_p=0.85)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# define your own sys & user msg:
sys_msg = "..."
user_msg = "..."
chat_msg = [
{"role": "system", "content": sys_msg},
{"role": "user", "content": user_msg}
]
input_text = tokenizer.apply_chat_template(chat_msg, tokenize=False, add_generation_prompt=False)
output = model.generate(input_text, sampling_params)
output_text = output[0].outputs[0].text
print(output_text) # show the model output