Downloads · 30 days
12
36% of all-time downloads
RichardErkhov/gpjt_-_Qwen1.5-0.5B-openassistant-guanaco-llama2-format-awq
gpjt_-_Qwen1.5-0.5B-openassistant-guanaco-llama2-format-awq is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
12
36% of all-time downloads
All-time downloads
33
Public
Parameters
464M
483 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors472 MB · 97%
How the weights are stored.
I32308M · 66%
From the Hugging Face model README
Quantization made by Richard Erkhov.
Qwen1.5-0.5B-openassistant-guanaco-llama2-format - AWQ
library_name: transformers datasets:
This is a fine-tune of Qwen/Qwen1.5-0.5B on the gpjt/openassistant-guanaco-llama2-format, which in turn is a version of timdettmers/openassistant-guanaco adjusted to use my best guess at the Llama 2 prompt format (see the dataset card for more info).
Sample code to use it:
import sys
import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
prompt_template = """
<s>[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>
{question} [/INST]
{response}
"""
def ask_question(model, tokenizer, question):
pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_new_tokens=2048)
prompt = prompt_template.format(question=question, response="")
tokens_in = len(tokenizer(prompt)["input_ids"])
start = time.time()
result = pipe(prompt)
end = time.time()
generated_text = result[0]['generated_text']
tokens_out = len(tokenizer(generated_text)["input_ids"])
print(generated_text)
tokens_generated = tokens_out - tokens_in
time_taken = end - start
tokens_per_second = tokens_generated / time_taken
print(f"{tokens_generated} tokens in {time_taken:.2f}s: {tokens_per_second:.2f} tokens/s)")
def test_model():
model_name = "Qwen1.5-0.5B-openassistant-guanaco-llama2-format"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cuda", torch_dtype=torch.bfloat16)
question = input("You: ")
ask_question(model, tokenizer, question)
if __name__ == "__main__":
test_model()