Qwen3.6-27B-oQ8e-fp16-mtp

Qwen3.6-27B-oQ8e-fp16-mtp

An oQ mixed-precision MLX quantization of Qwen/Qwen3.6-27B, built for Apple Silicon.

This is a dense model: every parameter is active on every token. It is markedly slower than the A3B MoE siblings on the same hardware -- see the benchmark note below before choosing it for long-context work.

Quantization

Uniform 8-bit affine, group size 64 (no per-module overrides).

Bit-width assignment is imatrix-guided (oqe_code_multilingual, 128 samples x 512 tokens) — 503 of 504 collected entries applied. Layers left unquantized are kept at fp16, which is what the fp16 in the repo name refers to. The mtp suffix means the checkpoint retains its multi-token-prediction head (1 MTP layer, shared embeddings), so MTP speculative decoding works without a separate draft model.

• Format: MLX safetensors, 30.9 GB across 2,209 tensors
• Mode: affine
• Calibration report: oq_imatrix_report.json is included in this repo

Architecture

This is a hybrid-attention model: only the 16 full-attention layers grow a KV cache with sequence length. The remaining 48 linear-attention layers hold a fixed-size recurrent state.

Measured performance

Single-stream, measured on an Apple M1 Max / 64 GB running oMLX. 300-token generations (median of 3) and a 7.5k-token uncached prefill.

These are throughput figures on one machine, not a quality benchmark, and they will vary with hardware, context length and sampler settings. The first request after a model load reads low — that is warm-up, not steady state.

Memory planning

At fp16 the KV cache costs roughly 67 KB per token (16 full-attention layers x 4 KV heads x 256 head dim x 2 for K/V x 2 bytes).

Budget for weights plus KV cache plus runtime overhead. If the cache does not fit, quantizing it (for example TurboQuant 8-bit) roughly halves these figures at no measurable throughput cost.

Usage

Served through oMLX over an OpenAI-compatible API:

curl http://127.0.0.1:8055/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen3.6-27B-oQ8e-fp16-mtp", "messages": [{"role": "user", "content": "Hello"}]}'

Sampling follows Qwen guidance: thinking mode temperature 0.6, top_p 0.95; non-thinking temperature 0.7, top_p 0.8; top_k 20, min_p 0 throughout. Avoid greedy decoding with thinking enabled — Qwen documents that the combination can produce endless repetition.

Verified with oMLX. Loading outside oMLX (mlx-lm / mlx-vlm) requires a runtime that honours per-module quantization overrides in config.json; this has not been tested here.

License

Apache 2.0, inherited from Qwen/Qwen3.6-27B. A copy of the license is included as LICENSE. Quantization does not change the licensing terms — all credit for the underlying model belongs to the Qwen team.