Downloads · 30 days
21
18% of all-time downloads
Star-gazer/Qwen3-Next-80B-A3B-Thinking-int4-ov
Qwen3-Next-80B-A3B-Thinking-int4-ov is a text generation model from Star-gazer. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Model creator: Qwen Original model: Qwen3-Next-80B-A3B-Thinking
Downloads · 30 days
21
18% of all-time downloads
All-time downloads
119
Public
Repo size
41.7 GB
Likes
0
Public
Click a slice to open those files.
.bin41.7 GB · 100%
From the Hugging Face model README
This is Qwen3-Next-80B-A3B-Thinking model converted to the OpenVINO™ IR (Intermediate Representation) format with weights compressed to INT4 by NNCF.
Weight compression was performed using nncf.compress_weights with the following parameters:
For more information on quantization, check the OpenVINO model optimization guide.
The provided OpenVINO™ IR model is compatible with:
pip install -U "git+https://github.com/huggingface/optimum-intel.git" torchvision "Pillow" --extra-index-url https://download.pytorch.org/whl/cpu
pip install --pre -U openvino --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
pip install -U "transformers==5.2"
import requests
from PIL import Image
from transformers import AutoProcessor
from optimum.intel.openvino import OVModelForVisualCausalLM
model_id = "Star-gazer/Qwen3-Next-80B-A3B-Thinking"
processor = AutoProcessor.from_pretrained(model_id)
model = OVModelForVisualCausalLM.from_pretrained(model_id)
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"
image = Image.open(requests.get(url, stream=True).raw)
messages = [
{
"role": "user",
"content": [
{"type": "image"},
{"type": "text", "text": "Describe this image."},
],
}
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])
For more examples and possible optimizations, refer to the Inference with Optimum Intel.
pip install huggingface_hub "Pillow"
pip install --pre -U openvino openvino-tokenizers openvino-genai --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
import huggingface_hub as hf_hub
model_id = "Star-gazer/Qwen3-Next-80B-A3B-Thinking-int4-ov"
model_path = "Qwen3-Next-80B-A3B-Thinking-int4-ov"
hf_hub.snapshot_download(model_id, local_dir=model_path)
import numpy as np
import openvino as ov
import openvino_genai as ov_genai
import requests
from PIL import Image
device = "CPU"
pipe = ov_genai.VLMPipeline(model_path, device)
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
image_tensor = ov.Tensor(np.array(image)[None])
print(pipe.generate("Describe this image.", image=image_tensor, max_new_tokens=200))
More GenAI usage examples can be found in OpenVINO GenAI library docs and samples
You can find more detaild usage examples in OpenVINO Notebooks:
1a. Deploy model on Windows using binary package:
ovms.exe --rest_port 8000 --source_model Star-gazer/Qwen3-Next-80B-A3B-Thinking-int4-ov --model_repository_path models --reasoning_parser qwen3 --tool_parser qwen3coder --target_device GPU --task text_generation
1b. Deploy model in a Docker container:
docker run -d --user $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) openvino/model_server:latest-gpu \
--rest_port 8000 --model_repository_path models --source_model Star-gazer/Qwen3-Next-80B-A3B-Thinking-int4-ov --reasoning_parser qwen3 --tool_parser qwen3coder --target_device GPU --task text_generation
pip install openai
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v3",
api_key="unused"
)
stream = client.chat.completions.create(
model="Star-gazer/Qwen3-Next-80B-A3B-Thinking-int4-ov",
messages=[{"role": "user", "content": "Hello."}],
stream=True,
extra_body={"chat_template_kwargs":{
"enable_thinking": True,
"preserve_thinking": True}},
tools=[],
)
for chunk in stream:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end="")
Also check how to use this model in an agentic flow with function calling, as shown in the agentic demo.
Note: This model supports preserve_thinking which greatly increases agentic capabilities.
Check the original model card for limitations.
The original model is distributed under Apache License Version 2.0 license. More details can be found in Qwen3-Next-80B-A3B-Thinking.
Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See Intel’s Global Human Rights Principles. Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.