webslate_-_gitai-awq

Quantization made by Richard Erkhov.

Github

Discord

Request more models

gitai - AWQ

• Model creator: https://huggingface.co/webslate/
• Original model: https://huggingface.co/webslate/gitai/

Original model description:

language:

• en license: apache-2.0 tags:
• chat
• code pipeline_tag: text-generation datasets:
• YashJain/GitAI library_name: transformers

---

YashJain/GitAI-Qwen2-0.5B-Instruct

Requirements

The code of Qwen2 has been in the latest Hugging face transformers and we advise you to install transformers>=4.37.0, or you might encounter the following error:

KeyError: 'qwen2'

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained(
    "YashJain/GitAI-Qwen2-0.5B-Instruct",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("YashJain/GitAI-Qwen2-0.5B-Instruct")

prompt = "How to undo my last commit"
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]