Qwopus3.5-9B-v3.5-gptq-int4

⚡ Qwopus3.5-9B-v3.5 — GPTQ Int4

Agentic 9B, squeezed into 8.45 GB. Runs on a 12 GB GPU.

GPTQ Int4 quantization of Jackrong/Qwopus3.5-9B-v3.5 — a multimodal tool-calling agent built on the qwen3_5 hybrid architecture (3:1 GatedDeltaNet : full attention + a 27-block vision tower).

🚀 Serve it (vLLM)

vllm serve malvavisc0/Qwopus3.5-9B-v3.5-gptq-int4 \
  --quantization gptq_marlin \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

The repo ships the full v22 chat template (chat_template.jinja) — vLLM picks it up automatically. It supports reasoning_effort, preserve_thinking, and tool_call_format (XML default, JSON optional).

On 12–16 GB cards, bound the KV cache:

  --max-model-len 8192 --max-num-seqs 10

🐍 Transformers

from transformers import AutoModelForMultimodalLM, AutoProcessor

model = AutoModelForMultimodalLM.from_pretrained(
    "malvavisc0/Qwopus3.5-9B-v3.5-gptq-int4", device_map="auto"
)
processor = AutoProcessor.from_pretrained(
    "malvavisc0/Qwopus3.5-9B-v3.5-gptq-int4"
)

🧪 How it was made

Quality comes from what you calibrate on — not from retraining:

• Format — GPTQ Int4, group_size=128, desc_act (activation ordering), symmetric. Compatible with vLLM's gptq_marlin kernel.
• Calibration — 128 agentic tool-use conversations from nvidia/Nemotron-Agentic-v1 (interactive_agent), each rendered through the shipped v22 chat template at 2048 tokens. Activation statistics match real serving traffic, not bare web text.
• Tooling — aft + gptqmodel 7.3.4, transformers 5.15.0, built on NVIDIA GB10 (sm_121).

Full build record in aft_provenance.json; per-layer quantization error in quant_log.csv.

🗂 What's quantized (and what isn't)

📌 Notes

• Source revision: dc2b00e1 (unchanged upstream since 2026-04).
• Tool calling uses the v22 XML format: <tool_call><function=name><parameter=key>value</parameter></function></tool_call>
• Reasoning renders in <think> blocks; effort is controllable via reasoning_effort (xhigh / medium / low).