Qwen3.5-4B-qat-lean-4bit-mlx

Qwen3.5-4B (Yooz QAT, uniform 4-bit)

What this is

The first grid-aligned quantization-aware-trained (QAT) build of Qwen3.5-4B for MLX. The 4-bit Qwen3.5 builds available today are post-training quantized (PTQ), and at this model size PTQ onto the MLX 4-bit grid breaks the model's aggregation abilities outright: on variable-tracking, the community 4-bit build scores 0.00 where this build scores 0.97 (same grid, same harness, n=30 -- full table below). NIAH-style retrieval looks fine on both, which is exactly why this damage goes unnoticed: simple retrieval does not exercise it.

This build is quantization-aware trained by Yooz Labs: the model is trained to tolerate exactly the MLX affine 4-bit group-64 grid via knowledge distillation from the full-precision bf16 model, so the quantized weights reproduce the original model's behavior -- aggregation AND multi-turn recall -- instead of trading one for the other, at both 8K and 32K context. It is a general-purpose build, not a task specialist.

It is also text-only and LEAN: the base checkpoint is natively multimodal (it ships a vision tower for image/video input), and the community 4-bit conversion carries that tower along unquantized in BF16 even though a text-only long-context workload never touches it. This build drops it -- every remaining tensor is 4-bit group-64, verifiable from config.json (no vision weights, no BF16 stragglers) -- which is the actual source of the size delta below, not a different quantization grid.

Provenance

• Base checkpoint: Qwen/Qwen3.5-4B
• QAT training: Yooz Labs, 2026-07 -- knowledge distillation from the bf16 base model under fake-quantization matching the target grid (4-bit, group size 64, affine). Training recipes are not published.
• Conversion tool: mlx-lm 0.31.3 (mlx_lm.convert, 4-bit group-64, uniform -- no per-tensor overrides)
• Harness: benchmarks/ (infinite repo, Yooz Labs), all numbers n=30 per cell on the machine noted below.

Size

Sizes are on-disk decimal GB (safetensors shard bytes). Until this build, getting working aggregation from Qwen3.5-4B on MLX meant the 8-bit build at 5.14 GB; this build delivers it at 2.37 GB.

Runtime memory (measured, not estimated)

Measured by the benchmark harness during real retrieval tasks on Apple M4 Pro (64 GB unified memory), macOS 26.5, mlx-lm 0.31.3, KV cache f16. Decode/prefill speed is near-identical between builds, as expected: QAT changes weight values, not the compute graph.

Validation

Long-context A/B against qwen3_5-4b-stock-text (verified identical quantization: affine 4-bit group-64, zero per-tensor overrides -- the community 4-bit build for this size uses a clean chat template, so its numbers match this comparator directly) on the same stack: needle-in-a-haystack (NIAH) and RULER-style tasks scored per-word, MRCR scored by sequence ratio. n=30 per cell.

Clean sweep at both context lengths: no task regresses relative to the same-grid PTQ baseline, aggregation is rescued at both bins, and multi-turn recall (MRCR) holds at parity with stock throughout -- this build actually posts its best variable-tracking score at 32K (0.99).

Usage

from mlx_lm import load, generate

model, tokenizer = load("YoozLabs/Qwen3.5-4B-qat-lean-4bit-mlx")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "List the first five prime numbers."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=64, verbose=True))

A GGUF (Q4_0) build of the same QAT weights is available at YoozLabs/Qwen3.5-4B-qat-GGUF for llama.cpp.

About Yooz Labs

Maintained by Yooz Labs -- sovereign, on-device AI for macOS. Everything we ship runs locally; nothing goes to a cloud. This build comes out of our long-context research line, where we found and quantified the PTQ aggregation crater -- and then trained it away.

Questions, issues, or corrections: [email protected]

License and attribution

Apache 2.0, inherited from the base checkpoint Qwen/Qwen3.5-4B. Qwen is a model family by Alibaba Cloud; this is an independent, unofficial QAT build, not distributed, endorsed, or certified by Alibaba.