Qwen3.5-9B-GGUF-Quant-Lab

Qwen3.5-9B GGUF Quant Lab

Reproducible llama.cpp GGUF quantization ladder for Qwen/Qwen3.5-9B, evaluated on English and Traditional Chinese perplexity, TMMLU+, throughput, and VRAM.

This repository contains the text-only language model. It does not include a vision projector (mmproj) and cannot provide the original model's image-input capability.

Recommended default

IQ4_XS-imatrix is the measured sweet spot on the RTX 4090 test system: 4.840 GiB, English/Traditional-Chinese PPL increases of +1.50%/+1.95% from F16, and a TMMLU+ change of -1.0 percentage point. Choose Q4_K_M-imatrix when you prefer slightly stronger PPL retention at a larger 5.243 GiB size.

Quantization and evaluation

Hardware: RTX 4090 24 GiB, Intel i7-13700, WSL2 Ubuntu, CUDA 13.1, llama.cpp b10031. Performance is hardware-specific. PPL is a next-token likelihood measure, not an instruction-following or safety score. TMMLU+ may have appeared in the base model's training data; here it is used only for relative quantization degradation because every quantization receives the identical fixed 500-question sample.

Full methodology, paired bootstrap CIs, Q2_K regression examples, figures, and reproducibility metadata are in EVAL_REPORT.md.

llama.cpp

./llama-cli -m Qwen3.5-9B-IQ4_XS-imatrix.gguf -ngl 99 --jinja --reasoning off -p "請用繁體中文介紹 GGUF 量化。"

Ollama

Download a GGUF file and the included Modelfile, then run:

ollama create qwen35-9b-quant-lab -f Modelfile
ollama run qwen35-9b-quant-lab

The supplied Modelfile points at Qwen3.5-9B-IQ4_XS-imatrix.gguf. Change FROM to select another local GGUF.

LM Studio

Download one .gguf file, open LM Studio's local model picker, and select the file. This is a Qwen3.5 text-only model; keep the embedded GGUF chat template enabled and disable reasoning if you want the same evaluation-style behavior.

Attribution and license

Base model: Qwen/Qwen3.5-9B at revision c202236235762e1c871ad0ccb60c8ee5ba337b9a. The original model and these quantized derivative weights are distributed under Apache-2.0; see LICENSE. Project scripts and documentation are MIT-licensed. This repository was staged for steven0226/Qwen3.5-9B-GGUF-Quant-Lab.