Downloads · 30 days
9
2% of all-time downloads
worldboss/TinyLlama-1.1B-chat-v0.3-awq
TinyLlama-1.1B-chat-v0.3-awq is a text generation model from worldboss. Use it when you need the model to write or continue text. It is set up for transformers.
Downloads · 30 days
9
2% of all-time downloads
All-time downloads
398
Public
Repo size
1.5 GB
Likes
1
Public
Click a slice to open those files.
.bin766 MB · 100%
From the Hugging Face model README
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer
quant_name = 'TinyLlama-1.1B-chat-v0.3-awq'
quant_path = "worldboss/{}".format(quant_name)
#quant_path = '/content/TinyLlama-1.1B-chat-v0.3-awq'
# Load model
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)
streamer = TextStreamer(tokenizer, skip_special_tokens=True)
# Convert prompt to tokens
prompt_template = """\
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
USER: {prompt}
ASSISTANT:"""
tokens = tokenizer(
prompt_template.format(prompt="How are you today?"),
return_tensors='pt'
).input_ids.cuda()
# Generate output
generation_output = model.generate(
tokens,
streamer=streamer,
max_new_tokens=512
)