Qwen3.8-27B-oQ4e-mtp

Qwen3.8-27B-oQ4e-mtp

Qwen/Qwen3.8-27B quantized for Apple Silicon with oQe (oQ enhanced quantization, oMLX v0.5.7). The format is MLX safetensors.

• Base precision: 4-bit affine, group size 64
• Effective size: 4.70 bits per weight over the language model, 4.89 over the whole repository
• Download: 17.0 GB in 4 shards
• Vision encoder: kept, in bfloat16 (0.92 GB, not quantized)
• MTP heads: kept and quantized (29 tensors)

[!IMPORTANT] This quantization was uploaded on 2026-08-14 and replaces a previous version. Re-download the weights if you pulled this repository before that date.

Variants

Sizes are decimal GB, as Hugging Face reports them. "bits/weight" covers the language model only, including the scales and biases of the quantized tensors. The bfloat16 vision encoder adds 0.92 GB to every variant.

Run it

oMLX

omlx serve gcoli/Qwen3.8-27B-oQ4e-mtp --port 8000

The server exposes an OpenAI-compatible API. The menu-bar app loads the same repository ID.

mlx-lm

pip install -U mlx-lm
mlx_lm.generate --model gcoli/Qwen3.8-27B-oQ4e-mtp --prompt "Explain affine quantization."

Unverified: mlx-lm support for the qwen3_5 vision-language architecture and for the MTP heads is not tested for this card. The language weights are standard MLX safetensors.

Sampling parameters

Qwen recommends these values for Qwen3.8-27B. The generation_config.json in this repository carries the thinking-mode values.

Do not use greedy decoding. It produces repetition in this model family.

Memory

Plan for about 24 GB of unified memory: the weights, the KV cache and the vision encoder share it. The context window is 262,144 tokens natively, so a long context adds several GB of KV cache. This is an estimate, not a measurement.

What oQe does

Standard affine quantization treats every weight in a group as equally important. oQe adds an importance matrix (imatrix): oMLX runs calibration inference, records the activation magnitude each weight actually sees, and weights the quantization error by that importance. Weights that drive large activations receive the smaller error.

oQ then allocates bits per tensor instead of per model. It measures the sensitivity of each tensor as MSE(float_output, quantized_output) / mean(float_output^2) and raises the bit width of the sensitive tensors above the base, while the total stays near the target.

Quantization recipe

These values are read back from the published config.json and oq_imatrix_report.json.

Calibration

Strict coverage was off, so the two unmatched tensors fell back to standard oQ affine quantization instead of failing the run. The calibration set is code and multilingual text. Expect the importance weighting to favour those domains over, for example, creative prose.

Limitations

• No benchmarks were run. This card reports no perplexity, no KL divergence and no task scores. The ordering in the variants table follows bit width, not measured quality.
• The vision encoder is unquantized, so image and video input costs the same memory in every variant.
• MTP heads only speed up generation on a runtime that implements multi-token prediction. On other runtimes they occupy disk and memory without an effect.
• Quantization below 4 bits degrades long-context and multi-step reasoning first. Test oQ3.5e-mtp on your own task before you rely on it.

Reproduce

Quantize the base model in oMLX with these options:

1. Enable oQe.
2. Enable Reuse imatrix cache, and leave the cache path empty.
3. Leave Strict imatrix coverage off.
4. Leave Text only off.
5. Enable Preserve MTP.
6. Set the non-quant dtype to bfloat16.
7. Select the oQ4 level and start the quantization.

License

Apache 2.0, inherited from Qwen/Qwen3.8-27B. The quantization changes the weights only. All model capabilities and restrictions come from the base model.

Credits

• Qwen team for Qwen3.8-27B.
• jundot/omlx for oMLX and the oQ quantization system.