Qwen3.5-9B-NF4DQ

Qwen3.5-9B NF4DQ

Qwen3.5-9B quantised to NF4DQ, 4.1562 bits per weight. 4.35 GiB.

This file requires a fork of llama.cpp. It will not load correctly in stock llama.cpp. See below.

Measured

Perplexity on wikitext-2 at context 512, A100-80GB, full test file:

At essentially the same size as Q3_K_M, perplexity is 9.4% better.

Throughput on an Apple M1 (8 GB), Metal:

The M1 has the narrowest memory bandwidth in the Apple Silicon line, so this is near the floor. CPU and Metal produce token-identical output at temperature 0.

This model has no multi-token-prediction head, so --spec-type draft-mtp does not apply.

Requires the NF4DQ fork

These weights use ggml type 43, which exists only in KayceeSamuel/llama.cpp (branch nf4dq). Stock llama.cpp does not support them.

Type 43 is currently the next free slot upstream. If upstream adds a quantisation type, a stock build will read these files as that type and decode garbage rather than refusing. If output looks wrong, check which build you are running before anything else.

git clone https://github.com/KayceeSamuel/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release   # omit -DGGML_CUDA on Apple Silicon
cmake --build build -j

Then:

./build/bin/llama-server -m MODEL.gguf -ngl 99 -c 8192 -fa on

What NF4DQ is

A 4.1562 bits-per-weight ggml block type: NF4 weight levels with a second quantisation stage that stores each 32-weight sub-block's scale as a 4-bit index into a fixed codebook rather than an fp16 value. That second stage is what takes it below Q4_K's 4.50 bpw.

Design notes, the fitting data behind every constant, and the approaches that were measured and rejected are in the fork's design doc.

Backends

CUDA and Metal have kernels. There is no Vulkan or SIMD CPU kernel, so AMD and Intel GPUs fall back to a scalar CPU path that is a correctness gate rather than a performance path.

Licence

Apache 2.0, matching the base model. Quantisation does not create a new work.