Downloads · 30 days
25
16% of all-time downloads
fsfsfsfsfsfsfsf24242/Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-v2-W4A16
Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-v2-W4A16 is a text generation model from fsfsfsfsfsfsfsf24242. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
4-bit weight quantization (compressed-tensors pack-quantized, W4A16) of Jackrong/Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-v2, made to run coherent + tool-calling on vLLM on a single consumer 12 GB GPU (e.g. RTX…
Downloads · 30 days
25
16% of all-time downloads
All-time downloads
159
Public
Parameters
9B
7.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors7.6 GB · 100%
How the weights are stored.
I326.9B · 77%
From the Hugging Face model README
4-bit weight quantization (compressed-tensors pack-quantized, W4A16) of
Jackrong/Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-v2,
made to run coherent + tool-calling on vLLM on a single consumer 12 GB GPU (e.g. RTX 3060).
Qwen3.5 is a GDN (Gated DeltaNet) hybrid architecture. On vLLM (Ampere), the quant format is decisive:
| format | result on vLLM |
|---|---|
compressed-tensors pack-quantized (this repo) | ✅ coherent, fits 12 GB (~7.2 GB) |
autoawq (quant_method=awq) | ❌ garbage output |
gptq / gptq_v2 (autogptq) | ❌ leaves GDN linear_attn layers fp16 → loads ~16.7 GB, won't fit 12 GB |
At publish time no Opus-distilled Qwen3.5-9B existed in this working format — hence this quant.
--gpu-memory-utilization 0.85.vllm serve <this-model> \
--dtype float16 --gpu-memory-utilization 0.85 --max-model-len 20480 --max-num-seqs 6 \
--enable-prefix-caching --mamba-cache-mode align \
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser hermes \
--trust-remote-code
Use
--tool-call-parser hermes— this distillate emits<tool_call>{json}</tool_call>(Hermes style). Withqwen3_xmlthe model still tool-calls correctly but vLLM can't parse it (emptytool_calls).
llmcompressor GPTQModifier(targets=["Linear"], scheme="W4A16", ignore=["lm_head"]).HuggingFaceH4/ultrachat_200k (seq len 1024).Quantization of the distillate by Jackrong, itself distilled from Claude-4.6-Opus reasoning onto Qwen3.5-9B. License inherits the base model's terms. Use responsibly.