Qwen3.5-4B-knapquant-GGUF

Qwen3.5-4B, per-tensor allocated GGUFs

Family: Qwen3.5-0.8B · Qwen3.5-4B

GGUF files of Qwen3.5-4B in which the type of every tensor is chosen by a budgeted search, then refined in place (GSQ) on the FFN. Each file has the size of an Unsloth Dynamic file of the same model, Unsloth's importance matrix and stock llama-quantize. Standard GGUF (llama.cpp, Ollama, LM Studio).

Recommended: Qwen3.5-4B-knapquant-4.6bpw.gguf.

Evaluation

This repository / the public file of the same size; llama.cpp 6c84c7d.

• KL: wikitext-2 test, 240 chunks of 512 tokens (llama-perplexity --kl-divergence).
• 5 tasks: mean of HellaSwag, Winogrande, ARC easy and challenge, MMLU (llama-perplexity).

These files replace the first release of this repository (allocation only, same sizes: KL 0.162 / 0.116 / 0.035 / 0.017).

Method

Every tensor is quantized at every type (IQ2_XXS to Q8_0) with Unsloth's importance matrix; a per-kind coefficient turns its error into a predicted KL; an exact knapsack picks one type per tensor under the byte budget. Then the FFN weights are refined in their GGUF format (GSQ, as in ISTA-DASLab's GSQ-RCO, same bytes). Type maps in type-maps/, code at github.com/MissingPackage/knapquant.

Usage

hf download MissingPackage/Qwen3.5-4B-knapquant-GGUF Qwen3.5-4B-knapquant-4.6bpw.gguf --local-dir .
llama-server -m Qwen3.5-4B-knapquant-4.6bpw.gguf -ngl 99 -c 32768

Thinking is off unless the request sets chat_template_kwargs: {"enable_thinking": true}. Text only: no vision projector.

License

apache-2.0, inherited from Qwen3.5-4B. The importance matrix is Unsloth's.