Qwen2-7B-Instruct-imatrix-GGUF

Quantizations of https://huggingface.co/Qwen/Qwen2-7B-Instruct

Note: you should use latest llama.cpp version with -fa switch to avoid garbage output.

From original readme

Requirements

The code of Qwen2 has been in the latest Hugging face transformers and we advise you to install transformers>=4.37.0, or you might encounter the following error:

KeyError: 'qwen2'

Quickstart

Here provides a code snippet with apply_chat_template to show you how to load the tokenizer and model and how to generate contents.

from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2-7B-Instruct",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-7B-Instruct")

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Processing Long Texts

To handle extensive inputs exceeding 32,768 tokens, we utilize YARN, a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts.

For deployment, we recommend using vLLM. You can enable the long-context capabilities by following these steps:

1. Install vLLM: You can install vLLM by running the following command.

pip install "vllm>=0.4.3"

Or you can install vLLM from source.

1. Configure Model Settings: After downloading the model weights, modify the config.json file by including the below snippet:

       {
           "architectures": [
               "Qwen2ForCausalLM"
           ],
           // ...
           "vocab_size": 152064,
   
           // adding the following snippets
           "rope_scaling": {
               "factor": 4.0,
               "original_max_position_embeddings": 32768,
               "type": "yarn"
           }
       }
   

   This snippet enable YARN to support longer contexts.

2. Model Deployment: Utilize vLLM to deploy your model. For instance, you can set up an openAI-like server using the command:

   python -m vllm.entrypoints.openai.api_server --served-model-name Qwen2-7B-Instruct --model path/to/weights
   

   Then you can access the Chat API by:

   curl http://localhost:8000/v1/chat/completions \
       -H "Content-Type: application/json" \
       -d '{
       "model": "Qwen2-7B-Instruct",
       "messages": [
           {"role": "system", "content": "You are a helpful assistant."},
           {"role": "user", "content": "Your Long Input Here."}
       ]
       }'
   

   For further usage instructions of vLLM, please refer to our Github.

Note: Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the rope_scaling configuration only when processing long contexts is required.