Qwen2.5-1.5B-Instruct-MLX-4bit-DWQ

Qwen2.5-1.5B-Instruct-MLX-4bit-DWQ

majentik's first DWQ pack. A 4-bit (group-size 32) MLX quantization of Qwen/Qwen2.5-1.5B-Instruct, refined with Distilled Weight Quantization (DWQ) — the quantization scales/biases are fine-tuned against an 8-bit teacher, recovering quality that plain affine 4-bit loses. Produced with the upstream MLX-LM learned-quant path (mlx_lm.dwq), no forks.

• Teacher: majentik/Qwen2.5-1.5B-Instruct-MLX-8bit (our own 8-bit pack; ≈16-bit quality)
• Method: DWQ, 4-bit, group-size 32, distilled from the 8-bit teacher
• Tooling: mlx-lm 0.31.3 (mlx 0.32.0)

Variants

<!-- sibling-table -->

Quality gate

Perplexity over a fixed text sample (lower is better); DWQ must be ≥ as good as the plain affine 4-bit sibling to ship public:

DWQ recovers ≈27% of the 4-bit→teacher gap. Training KL loss fell 0.116 → 0.050. A 5-prompt greedy sanity check (reasoning, translation, code, arithmetic) is coherent with no loops. Gate: PASS.

Exact gate invocation (mlx-lm 0.31.3, deterministic — no dataset, no sampling, no seed; single forward pass, mean token NLL over a fixed passage, 197 target tokens; ppl = exp(nll); full script in the library's docs/dwq-recipe.md "Gate: exact invocation"):

import mlx.core as mx; from mlx_lm import load
m, t = load("majentik/Qwen2.5-1.5B-Instruct-MLX-4bit-DWQ")
ids = mx.array(t.encode(TEXT))[None]  # TEXT = fixed passage from docs/dwq-recipe.md
lg = m(ids[:, :-1]).astype(mx.float32)
lp = lg - mx.logsumexp(lg, axis=-1, keepdims=True)
nll = -mx.take_along_axis(lp, ids[:, 1:][..., None], axis=-1).mean()
print(float(mx.exp(nll)))  # -> 9.45

The gate condition is the ordering (DWQ ≤ plain affine 4-bit); absolute perplexity values vary with the sample passage.

Recipe

# dwq_stable.py — forces the stable reference KL path (see repo docs/dwq-recipe.md)
import mlx_lm.tuner.losses as _losses
_losses.can_run_metal = lambda: False
from mlx_lm.quant.dwq import main
if __name__ == "__main__":
    main()

python dwq_stable.py \
  --model Qwen2.5-1.5B-Instruct-MLX-8bit \
  --quantized-model Qwen2.5-1.5B-Instruct-MLX-4bit \
  --mlx-path Qwen2.5-1.5B-Instruct-MLX-4bit-DWQ \
  --group-size 32 --bits 4 \
  --num-samples 1024 --max-seq-length 512 --batch-size 1 \
  --learning-rate 1e-6 --data-path allenai/tulu-3-sft-mixture

Full reproduction (teacher/baseline build, gate, and the mlx-lm 0.31.3 Metal-KL workaround, filed upstream as ml-explore/mlx-lm#1700) is documented in the library's docs/dwq-recipe.md, per docs/quantization-policy.md rule 2 (DWQ for 2–4-bit packs; group-size 32).

Usage

pip install mlx-lm
python -m mlx_lm generate --model majentik/Qwen2.5-1.5B-Instruct-MLX-4bit-DWQ \
  --prompt "Explain what a hash table is in two sentences."

License

Apache-2.0, inherited from the base model Qwen/Qwen2.5-1.5B-Instruct.