Qwen3.5-0.8B-qat-lean-4bit-mlx

Qwen3.5-0.8B (Yooz QAT, uniform 4-bit)

Context limit: 8K. This build is validated healthy up to 8,192 tokens. Beyond that (tested to 32K) it regresses on multi-hop retrieval -- NIAH single-needle drops to 0.03 accuracy, below the untrained 4-bit baseline. Do not use this build past 8K context. If you need long context at this size class, use Qwen3.5-4B-qat-lean-4bit-mlx instead, which is clean through 32K. Full breakdown in Validation below.

What this is

The first grid-aligned quantization-aware-trained (QAT) build of Qwen3.5-0.8B for MLX. The 4-bit Qwen3.5 builds available today are post-training quantized (PTQ), and at this model size PTQ onto the MLX 4-bit grid damages the model's aggregation abilities: on variable-tracking at 8K, the community 4-bit build scores 0.23 where this build scores 0.78 (same grid, same harness, n=30 -- full table below). NIAH-style retrieval looks fine on both at 8K, which is exactly why this damage goes unnoticed: simple retrieval does not exercise it.

This build is quantization-aware trained by Yooz Labs: the model is trained to tolerate exactly the MLX affine 4-bit group-64 grid via knowledge distillation from the full-precision bf16 model, so the quantized weights reproduce the original model's behavior instead of trading one task for another. It is a general-purpose build, not a task specialist.

It is also text-only and LEAN: the base checkpoint is natively multimodal (it ships a vision tower for image/video input), and the community 4-bit conversion carries that tower along unquantized in BF16 even though a text-only long-context workload never touches it. This build drops it -- every remaining tensor is 4-bit group-64, verifiable from config.json (no vision weights, no BF16 stragglers) -- which is the actual source of the size delta below, not a different quantization grid.

Provenance

• Base checkpoint: Qwen/Qwen3.5-0.8B
• QAT training: Yooz Labs, 2026-07 -- knowledge distillation from the bf16 base model under fake-quantization matching the target grid (4-bit, group size 64, affine). Training recipes are not published.
• Conversion tool: mlx-lm 0.31.3 (mlx_lm.convert, 4-bit group-64, uniform -- no per-tensor overrides)
• Harness: benchmarks/ (infinite repo, Yooz Labs), all numbers n=30 per cell on the machine noted below.

Size

Sizes are on-disk decimal GB (safetensors shard bytes). Until this build, getting working aggregation from Qwen3.5-0.8B on MLX meant the 8-bit build at 1.00 GB; this build delivers it at 0.42 GB, for use up to 8K.

Runtime memory (measured, not estimated)

The 32K rows are for size/speed reference only -- see the context limit above, this build should not be served at 32K in practice. Measured by the benchmark harness during real retrieval tasks on Apple M4 Pro (64 GB unified memory), macOS 26.5, mlx-lm 0.31.3, KV cache f16. Decode/prefill speed is near-identical between builds, as expected: QAT changes weight values, not the compute graph.

Validation

Long-context A/B against qwen3_5-0.8b-stock-text (verified identical quantization: affine 4-bit group-64, zero per-tensor overrides -- the same grid, minus the community build's unrelated template defect where it leaks its thinking preamble into exact-match tasks) on the same stack: needle-in-a-haystack (NIAH) and RULER-style tasks scored per-word, MRCR scored by sequence ratio. n=30 per cell.

Read the 32K row as a regression, not noise. This build was trained with a short sequence window, and past 8K it falls at or below the untrained same-grid baseline on four of five tasks, catastrophically on NIAH single-needle. The 8K numbers above are real and reproducible; the 32K numbers are a documented failure mode. If your workload can exceed 8K tokens, do not use this build -- use the 4B QAT build, or the plain 4-bit/8-bit community PTQ builds, both of which stay coherent (if not aggregation-healthy) at 32K.

Usage

from mlx_lm import load, generate

model, tokenizer = load("YoozLabs/Qwen3.5-0.8B-qat-lean-4bit-mlx")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "List the first five prime numbers."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=64, verbose=True))

A GGUF (Q4_0) build of the same QAT weights is available at YoozLabs/Qwen3.5-0.8B-qat-GGUF for llama.cpp.

About Yooz Labs

Maintained by Yooz Labs -- sovereign, on-device AI for macOS. Everything we ship runs locally; nothing goes to a cloud. This build comes out of our long-context research line, where we found and quantified the PTQ aggregation crater -- and then trained it away.

Questions, issues, or corrections: [email protected]

License and attribution

Apache 2.0, inherited from the base checkpoint Qwen/Qwen3.5-0.8B. Qwen is a model family by Alibaba Cloud; this is an independent, unofficial QAT build, not distributed, endorsed, or certified by Alibaba.