Qwen3.8-27B-MLX-4bit-DWQ

Qwen3.8-27B MLX 4-bit DWQ

This is a 4-bit MLX derivative of Qwen/Qwen3.8-27B, calibrated for tool calling, agentic software-engineering traces, multilingual chat, and Python code.

The language tower was trained with Distilled Weight Quantization (DWQ) against the pinned 8-bit MLX teacher. The vision tower was not DWQ-trained: its 333 tensors are copied exactly from the pinned mlx-community 4-bit RTN conversion. For Qwen3.8's MTP path, use the separately loadable official MLX drafter mlx-community/Qwen3.8-27B-MTP-4bit, pinned here at revision b643c01b6d3b094e325edb6ebd832e16c486c575.

What changed

The final language checkpoint contains 1,847 tensors; the complete VLM contains 2,180 tensors (1,847 language + 333 vision).

Results

The primary metric is KL divergence to sparse top-1,024 logits from the pinned 8-bit teacher on a fixed, disjoint 64-example holdout. Lower is better.

The final KL is 63.596% lower than the baseline. The predeclared acceptance threshold was 0.203890.

A deterministic 11-prompt smoke suite produced the following results with thinking disabled:

On an Apple M2 Max with 64 GB unified memory, the DWQ model averaged 67.68 prompt tok/s and 23.19 generation tok/s with a 16.05 GB peak in this small suite. The RTN baseline measured 67.96 and 23.16 tok/s with the same peak. This is a compatibility smoke test, not a broad capability benchmark.

The pinned official MTP drafter also passed a three-prompt, 256-token greedy smoke against this DWQ target with 3/3 outputs exactly matching autoregressive decoding. It accepted 83.8% of drafted tokens and improved mean generation throughput by 16.1% in that small test; peak MLX memory changed from 18.94 to 19.72 GB. Speedup is prompt-dependent, and one prompt was slightly slower with MTP.

Calibration mix

The 1,024 training and 64 validation examples are source-disjoint. Rendered length is capped at 513 tokens.

The multilingual slice covers Russian, Chinese, Ukrainian, Polish, Standard Arabic, Spanish, and Japanese, with double weight on Russian. Only aggregate provenance is published; source rows and teacher logits are not redistributed. See CALIBRATION.md and calibration-manifest.json.

Usage

Text-only generation with MLX-LM:

pip install -U mlx-lm
mlx_lm.generate \
  --model WaveCut/Qwen3.8-27B-MLX-4bit-DWQ \
  --prompt "Как называется столица Польши? Ответь одним словом." \
  --max-tokens 32 \
  --temp 0 \
  --chat-template-config '{"enable_thinking": false}'

Image-text generation with MLX-VLM:

pip install -U mlx-vlm
mlx_vlm.generate \
  --model WaveCut/Qwen3.8-27B-MLX-4bit-DWQ \
  --image /path/to/image.png \
  --prompt "Describe this image." \
  --max-tokens 128 \
  --temperature 0

Speculative decoding with the official MTP drafter:

pip install -U "mlx-vlm>=0.6.8"
mlx_vlm.generate \
  --model WaveCut/Qwen3.8-27B-MLX-4bit-DWQ \
  --draft-model mlx-community/Qwen3.8-27B-MTP-4bit \
  --prompt "Write a robust retrying HTTP client in Python." \
  --max-tokens 256 \
  --temperature 0

--draft-kind mtp is detected automatically. The drafter is separate because MLX-VLM binds it to the target model's embeddings and language-model head at runtime. This DWQ release does not redistribute or claim authorship of the MTP weights.

The release was built and clean-loaded with mlx-lm==0.31.3, mlx==0.32.0, mlx-vlm==0.6.8, Python 3.12.9, and uv==0.12.3.

Reproducibility

The exact revisions, hyperparameters, corpus hashes, validation curve, local MLX-LM compatibility patch, and environment lock are included in RECIPE.md, calibration-manifest.json, evaluation-summary.json, mlx-lm-0.31.3-local-dwq-data.patch, and uv.lock.

Limitations

• DWQ optimization covered the language tower only; vision quality is that of the upstream 4-bit RTN conversion and was only clean-load-smoked here.
• The calibration corpus and behavioral smoke suite are small relative to the model's full capability surface. No claim is made for broad benchmark gains.
• Tool-call behavior depends on the supplied tool schema and Qwen chat template. Validate formats required by your runtime.
• The external MTP tensors come from the original Qwen3.8 checkpoint and were not DWQ-retuned. Greedy compatibility with the pinned official MLX drafter is verified, but acceptance and speedup are workload-dependent.
• Quantization can change outputs. Evaluate safety, multilingual quality, long-context behavior, and task-specific reliability before deployment.

The original model and this derivative are released under Apache-2.0. See LICENSE and THIRD_PARTY_NOTICES.md.