Downloads · 30 days
0
Vicenterivera/qwen3.5-35b-a3b-int8-onnx-32k
qwen3.5-35b-a3b-int8-onnx-32k is a machine learning model from Vicenterivera. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for onnxruntime. The card lists the license as apache-2.0.
ONNX-Runtime-GenAI-format edge build of QWEN3.5-35B-A3B with int8-fp16 quantisation, 32768-token context window.
Downloads · 30 days
0
Access
Public
Updated May 26, 2026
Repo size
35.3 GB
Likes
0
Public
Click a slice to open those files.
.bin35.3 GB · 100%
From the Hugging Face model README
ONNX-Runtime-GenAI-format edge build of QWEN3.5-35B-A3B with int8-fp16 quantisation, 32768-token context window.
| Metric | Value |
|---|---|
| Family | qwen3.5 |
| Quantisation | int8-fp16 |
| Context length | 32,768 tokens |
| Decode rate (GB10, single stream) | ~11 tok/s |
| Source build | github.com/vriveras/qwen-3-5-ort-support @ 38bd75ae |
| Source dir | Qwen3.5-35B-A3B-NoRot-GPTQ-Int8-FP16-ONNX-GenAI-32K |
model.onnx — ONNX graph topology. Uses custom ops in domain
ai.ort.contrib.ssm for the SSM linear-attention layers
((via stock ORT MatMulNBits for attention/MoE)).weights.bin — external initializer blob referenced by model.onnx.genai_config.json — ORT GenAI runtime config (input/output names,
KV + SSM cache pair declarations, custom_ops_library path).tokenizer.json + tokenizer_config.json + chat_template.jinja —
HuggingFace-style tokenizer + chat template.This model needs:
qwen3-ssm-cuda-graph, or apply the patches from the source
repo's genai_patches/ against upstream microsoft/onnxruntime-genai.libgated_delta_net_op.{so,dll} built from
github.com/vriveras/qwen-3-5-ort-support
(bash install.sh on Linux, .\build_windows.ps1 on Windows).genai_config.json's
model.decoder.session_options.custom_ops_library to point at where
you installed the custom op library.git lfs install
git clone https://huggingface.co/USER/qwen3.5-35b-a3b-int8-onnx-32k /path/to/models/Qwen3.5-35B-A3B-NoRot-GPTQ-Int8-FP16-ONNX-GenAI-32K
# Edit genai_config.json's custom_ops_library path to match your install
cd /path/to/your/clone-of/qwen-3-5-ort-support
bash start_server.sh --genai --model_dir /path/to/models/Qwen3.5-35B-A3B-NoRot-GPTQ-Int8-FP16-ONNX-GenAI-32K \
--tokenizer /path/to/models/Qwen3.5-35B-A3B-NoRot-GPTQ-Int8-FP16-ONNX-GenAI-32K
Or with the multi-model server:
mkdir /path/to/models && \
git clone https://huggingface.co/USER/qwen3.5-35b-a3b-int8-onnx-32k /path/to/models/Qwen3.5-35B-A3B-NoRot-GPTQ-Int8-FP16-ONNX-GenAI-32K
python serve_genai.py --models_dir /path/to/models
The server auto-discovers the model and registers it under the Ollama
canonical name qwen3.5:int8-fp16-32k (or just qwen3.5 for
the default tag).
Apache 2.0, inherits from the upstream Qwen3.x model weights.