Qwen3.8-4B-Distill-GGUF-Q2_K

Qwen3.8-4B — GGUF (Q2_K)

Q2_K imatrix quantization of empero-ai/Qwen3.8-4B — a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture — for llama.cpp, Ollama, LM Studio, Jan, KoboldCpp, and other stock GGUF runtimes.

This repo hosts the Q2_K quant only. For higher-quality quantizations (Q4_K_M and up), see empero-ai/Qwen3.8-4B-Distill-GGUF. Capability details and full benchmark results live on the main model card.

[!Note] Qwen3.5-class models are hybrids: three Gated DeltaNet layers for every full-attention layer. A recent llama.cpp build with Qwen3.5 / Gated DeltaNet support is required — older builds will fail to load the architecture.

[!Warning] Q2_K is an aggressive 2-bit quantization. Expect a noticeable quality drop compared to Q4_K_M and above, especially on long reasoning chains. Use this file when memory is the hard constraint.

Files

Size is exact decimal GB from the uploaded file (1 GB = 1,000,000,000 bytes).

Quantization details

• Method: Q2_K with importance matrix (imatrix) weighting
• Calibration: imatrix.dat computed from a diverse English calibration dataset (~4,900 chunks)
• Source weights: Qwen3.8-4B-BF16.gguf (full-precision reference)

Usage

llama.cpp

llama-cli -m Qwen3.8-4B-Q2_K.gguf \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -n 16384 -cnv

Use the built-in chat template (-cnv). The model is a reasoning model: every answer opens with a <think> block, so allow a generous -n and strip the <think>...</think> span for end users.

Ollama / LM Studio / Jan / KoboldCpp

Download the GGUF and load it directly; the chat template is embedded in the file. Recommended sampling: temperature=0.6, top_p=0.95, top_k=20.

Model architecture (from GGUF metadata)

Provenance & licensing

Quantization of empero-ai/Qwen3.8-4B, a distillation of Qwen3.8 2.4T A95B into Qwen/Qwen3.5-4B.

This repo's weights are distributed under a custom license (see LICENSE): free for personal / non-commercial use; commercial use requires a paid license — contact [email protected]. The underlying source weights remain available under their original Apache-2.0 terms from the upstream repositories above.

Acknowledgements

• Source model: Empero
• Base model: Qwen3.5-4B (Alibaba Qwen team)
• GGUF quantization: llama.cpp (ggml-org)