stablelm-zephyr-3b-openvino-4bit

This model is a quantized version of stabilityai/stablelm-zephyr-3b and is converted to the OpenVINO format. This model was obtained via the nncf-quantization space with optimum-intel.

Please note: For commercial use, please refer to https://stability.ai/license.

Model Description

StableLM Zephyr 3B is a 3 billion parameter instruction tuned inspired by HugginFaceH4's Zephyr 7B training pipeline this model was trained on a mix of publicly available datasets, synthetic datasets using Direct Preference Optimization (DPO), evaluation for this model based on MT Bench and Alpaca Benchmark

Model Parameters

context window   = 4096
model type       = 3B
model params     = 2.80 B
BOS token        = 0 '<|endoftext|>'
EOS token        = 0 '<|endoftext|>'
UNK token        = 0 '<|endoftext|>'
PAD token        = 0 '<|endoftext|>'

The tokenizer of this model supports chat_templates

Usage

StableLM Zephyr 3B uses the following instruction format:

<|user|>
List 3 synonyms for the word "tiny"<|endoftext|>
<|assistant|>
1. Dwarf
2. Little
3. Petite<|endoftext|>

Model Details

• Developed by: Stability AI
• Model type: StableLM Zephyr 3B model is an auto-regressive language model based on the transformer decoder architecture.
• Language(s): English
• Library: Alignment Handbook
• Finetuned from model: stabilityai/stablelm-3b-4e1t
• License: StabilityAI Community License.
• Commercial License: to use this model commercially, please refer to https://stability.ai/license
• Contact: For questions and comments about the model, please email [email protected]

First make sure you have optimum-intel installed:

pip install openvino-genai==2024.4.0
pip install optimum-intel[openvino]

To load your model you can do as follows:

from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer, AutoConfig
from threading import Thread
from transformers import TextIteratorStreamer

model_id = "FM-1976/stablelm-zephyr-3b-openvino-4bit"
model = OVModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
ov_model = OVModelForCausalLM.from_pretrained(
    model_id = model_id,
    device='CPU',
    ov_config={"PERFORMANCE_HINT": "LATENCY", "NUM_STREAMS": "1", "CACHE_DIR": ""},
    config=AutoConfig.from_pretrained(model_id)
)


# Generation with a prompt message
question = 'Explain the plot of Cinderella in a sentence.'
messages = [
    {"role": "user", "content": question}
]

print('Question:', question)
#Credit to https://github.com/openvino-dev-samples/chatglm3.openvino/blob/main/chat.py
streamer = TextIteratorStreamer(tokenizer, timeout=60.0, skip_prompt=True, skip_special_tokens=True)
model_inputs = tokenizer.apply_chat_template(messages,
                                                     add_generation_prompt=True,
                                                     tokenize=True,
                                                     pad_token_id=tokenizer.eos_token_id,
                                                     num_return_sequences=1,
                                                     return_tensors="pt")
generate_kwargs = dict(input_ids=model_inputs,
                        max_new_tokens=450,
                        temperature=0.1,
                        do_sample=True,
                        top_p=0.5,
                        repetition_penalty=1.178,
                        streamer=streamer)
t1 = Thread(target=ov_model.generate, kwargs=generate_kwargs)
t1.start()
for new_text in streamer:
    new_text = new_text
    print(new_text, end="", flush=True)