Qwen3.5-9bxOpus-4.7-sonic-GGUF

Qwen3.5-9bxOpus-4.7-sonic-GGUF : GGUF

A distilled 9B code/reasoning model in GGUF format, optimized for local inference via llama.cpp.

---

Quick Start

llama-cli -hf nphearum/Qwen3.5-9bxOpus-4.7-sonic-GGUF --jinja

---

Server Launch

llama-server \
  -m Qwen3.5-9bxOpus-4.7-sonic-GGUF.Q4_K_M.gguf \
  --port 8001 --alias qwen3.5-9b-opus \
  -c 65536 -n 8192 \
  --temp 0.6 --top-p 0.95 --top-k 40 --repeat-penalty 1.05 \
  --flash-attn on --ctk q8_0 --ctv q8_0 \
  --jinja --chat-template-kwargs '{"enable_thinking": true}' \
  -ngl -1

Recomment configure

Using mmproj is more like to Multi-Token Prediction(MTP) -- reason for support text, image, video, audio and pdf, is an advanced training and inference technique for memory and cache. I recomment to use --ctk q8_0 --ctv q8_0 for better cache and load.

llama-server \
  -m Qwen3.5-9bxOpus-4.7-sonic-GGUF.Q4_K_M.gguf \
  --port 53281 \
  -c 131072 \
  --parallel 1 \
  --flash-attn on \
  --no-context-shift \
  -ngl -1 \
  --jinja \
  --chat-template-kwargs "{\"enable_thinking\": true}" \
  --mmproj Qwen3.5-9bxOpus-4.7-sonic-GGUF.BF16-mmproj.gguf

---

Preset Configs

---

Key Specs

• Base: Qwen3.5 9B · Format: Q4_K_M GGUF
• Context: 32K–64K practical · Output: up to 8K tokens
• KV cache: 8-bit (q8_0) · GPU: full offload (-ngl -1)
• Thinking mode: optional — improves reasoning, adds latency

---

Caveats

• Quality drops beyond 64K context
• 9B class — sensitive to sampling parameters
• Thinking mode can produce unstable output if misaligned with training