Downloads · 30 days
11
3% of all-time downloads
Mihaiii/stablelm-zephyr-3b-OV_FP14-4BIT
stablelm-zephyr-3b-OV_FP14-4BIT is a text generation model from Mihaiii. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
The quantized version of stablelm-zephyr-3b after running the steps on from here
Downloads · 30 days
11
3% of all-time downloads
All-time downloads
427
Public
Repo size
1.6 GB
Likes
0
Public
Click a slice to open those files.
.bin1.6 GB · 100%
From the Hugging Face model README
The quantized version of stablelm-zephyr-3b after running the steps on from here
You can use it like this (steps taken from the above link):
pip install -q git+https://github.com/huggingface/optimum-intel.git@e22a2ac26b3a6c7854da956d538f784ebeca879b onnx openvino-nightly
then
from optimum.intel.openvino import OVModelForCausalLM
from transformers import AutoConfig, AutoTokenizer
from optimum.utils import NormalizedTextConfig, NormalizedConfigManager
NormalizedConfigManager._conf['stablelm_epoch'] = NormalizedTextConfig.with_args(num_layers='num_hidden_layers', num_attention_heads='num_attention_heads')
NormalizedConfigManager._conf['stablelm-epoch'] = NormalizedTextConfig.with_args(num_layers='num_hidden_layers', num_attention_heads='num_attention_heads')
model_path = 'Mihaiii/stablelm-zephyr-3b-OV_FP14-4BIT'
model = OVModelForCausalLM.from_pretrained(model_path, compile=False, config=AutoConfig.from_pretrained(model_path, trust_remote_code=True), stateful=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
prompt = [{'role': 'user', 'content': 'List 3 synonyms for the word "tiny"'}]
inputs = tokenizer.apply_chat_template(
prompt,
add_generation_prompt=True,
return_tensors='pt'
)
tokens = model.generate(
inputs.to(model.device),
max_new_tokens=1024,
temperature=0.8,
do_sample=True
)
print(tokenizer.decode(tokens[0], skip_special_tokens=False))