Qwen3.8-27B-oQ4e-DWQ-MTP-Vision-MLX

Qwen3.8-27B-oQ4e-DWQ-MTP-Vision-MLX

Compressed for Apple Silicon

A benchmark-aligned oQ4e-XL (imatrix) → M4Q → DWQ build of Qwen3.8-27B. M4Q + DWQ reduced held-out teacher divergence by 20% (0.0287 → 0.0229) over the M4Q initializer and by 60% over the raw oQ4e-XL donor (0.0579), while preserving the original bf16 vision tower and native MTP speculative-decoding head.

A mixed-precision 4-bit MLX quantization of Qwen/Qwen3.8-27B — vision + text, with the original vision tower preserved in bf16 and the donor's mixed-precision MTP head retained for multimodal inference and native speculative decoding in oMLX.

⚠️ These are quantized weights. The model's capability comes from Qwen's base model — please star/cite it first. This repo contributes the oQ/DWQ quantization recipe, Hessian-guided M4Q protection and MTP/vision packaging; M4Q and DWQ tune the quantizer's scales/biases and do not add new knowledge.

Model lineage

Qwen/Qwen3.8-27B                          (Apache-2.0 · dense 27B · hybrid attention · vision + MTP)
  └─ THIS REPO: oQ4e-XL (imatrix) → M4Q → DWQ   + MTP + vision

• Architecture: dense Qwen3.8-27B (qwen3_5 MLX architecture), 64 hybrid Gated-DeltaNet/full-attention layers.
• Vision tower: Qwen3.8 vision encoder, kept in bf16 and packaged as 333 vision_tower.* tensors.
• MTP head: native multi-token prediction head for oMLX's speculative decoding.

Quantization: oQ4e-XL (imatrix) → M4Q → single-stage DWQ

This is not a plain round-to-nearest 4-bit conversion:

1. oQ4e-XL — importance-aware mixed precision. oMLX's enhanced quantizer builds an importance matrix from 256 × 512-token calibration samples (oqe_code_multilingual, adaptive) and allocates additional precision to sensitive tensors. The XL recipe then surgically promotes the architecture's precision-critical tensors to 8-bit: all linear_attn.* (Gated-DeltaNet) and self_attn.* projections, the embeddings and the lm_head — the same layer-swap idea as leonsarmiento/Qwen3.6-35B-A3B-oQ4e-mtp-XL-mlx. The result is an affine 4-bit, group size 64 base with 310 quantized modules promoted to 8-bit and 27 modules to 5-bit.
2. M4Q — Hessian-guided protection. The bf16 base is streamed and every module's affine quantization (scales/biases, group size 64, 4-bit base) is optimized against activation-weighted Hessian objectives from the calibration data, keeping the XL bit map.
3. DWQ — activation-aligned distillation. The trainable affine scales/biases of all sub-8-bit language modules are optimized toward an oQ8e teacher made from the same base model. The objective is KL divergence over the teacher's top-1024 logits at temperature 2.0. Training uses batch 1, 512-token windows, gradient checkpointing, Adam, a cosine LR schedule, validation early stop and exports only the best checkpoint.
4. Component splice. The donor's mixed-precision MTP head and bf16 vision tower are copied back byte-for-byte after DWQ; neither is trained by the text-only DWQ pass.

The loss above measures fidelity to this recipe's 8-bit teacher on its held-out calibration distribution. It is not task accuracy and should not be compared with losses from another dataset, tokenizer, teacher or sequence length. The oQ8e teacher itself sits at KL 0.019 vs the bf16 base, so the final model (0.023) is within ~20% of the teacher's own noise budget.

Calibration mix

The single DWQ stage retains general reasoning while emphasizing the model's intended code and agent workloads. DWQ sees tokenized activation windows, not additional SFT updates.

Agent + code data is 62.5% of the mix. The DWQ calibration itself is text-only; multimodal capability comes from the unmodified bf16 vision tower and the distilled language backbone.

Evaluation

Quantization fidelity measured on this build

No task or vision-benchmark score is claimed here yet. The result above demonstrates improved language-logit fidelity on the held-out calibration split, not guaranteed benchmark improvement. (The raw-donor number was measured on a 16-batch subset; the M4Q/DWQ numbers on the full 128-batch split.)

Repos in this family

The siblings share the same DWQ language backbone and MTP head (backbone shards are hard-linked); this repo additionally contains the original bf16 vision tower.

How to run

These are MLX weights for Apple Silicon. The tested serving path is oMLX, which supports Qwen3.8, vision input and native MTP speculative decoding.

# Download directly into the oMLX model directory
hf download Litwein/Qwen3.8-27B-oQ4e-DWQ-MTP-Vision-MLX \
  --local-dir ~/.omlx/models/Qwen3.8-27B-oQ4e-DWQ-MTP-Vision-MLX

# Enable native MTP once
curl -X PUT \
  http://127.0.0.1:8003/admin/api/models/Qwen3.8-27B-oQ4e-DWQ-MTP-Vision-MLX/settings \
  -H "Authorization: Bearer sk-local" -H "Content-Type: application/json" \
  -d '{"mtp_enabled": true}'

# OpenAI-compatible multimodal API
curl -X POST http://127.0.0.1:8003/v1/chat/completions \
  -H "Authorization: Bearer sk-local" -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.8-27B-oQ4e-DWQ-MTP-Vision-MLX",
       "messages":[{"role":"user","content":[
         {"type":"image_url","image_url":{"url":"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}},
         {"type":"text","text":"Describe the image and explain the visual clues."}
       ]}],
       "max_tokens":4096,"temperature":1.0,"top_p":0.95}'

The oMLX model id is case-sensitive and matches the downloaded folder name. Native MTP and the multimodal packaging require a compatible runtime; oMLX is the path validated for this release.

Recommended sampling

Follow the base model's recommended settings (Qwen3-style defaults: thinking enabled, temperature ~0.7, top_p ~0.8; greedy decoding is useful for reproducible benchmarks). Give hard reasoning/code tasks a generous output budget (32k or more where practical).

Intended use & limitations

• Best suited to: multimodal reasoning, visual question answering, coding, tool/agent workflows, math, STEM and long-context chat.
• Quantization is lossy: for maximum fidelity use the bf16 base or a higher-bit quant.
• DWQ is calibration, not SFT: it improves quantized-teacher fidelity on represented activations; it does not teach facts or guarantee gains on every benchmark.
• Vision was preserved, not vision-calibrated: the bf16 tower is unchanged, but this DWQ run used text calibration and has not been assigned a measured vision-benchmark score.
• Calibration coverage: public resolved SWE trajectories are Python-centric; multilingual SWE/terminal/skills behaviour was not directly calibrated.
• Long context costs memory: the architectural context maximum is not a promise that every Apple Silicon machine can allocate the corresponding KV/cache state.

Acknowledgements

• Qwen team — Qwen3.8-27B (Apache-2.0).
• Apple MLX — mlx, mlx-lm, mlx-vlm and mlx_lm.quant.dwq.
• oMLX — enhanced oq/oQe imatrix quantization, M4Q support and MTP + vision serving runtime.
• Calibration-data authors — SWE-bench/SWE-smith, OpenThoughts, Open-R1, BigCode, NVIDIA OpenCodeReasoning and AllenAI Tulu.

License

Apache-2.0, inherited from Qwen/Qwen3.8-27B.

Citation

Please cite the original base model:

@misc{qwen3.8-27b,
  title  = {Qwen3.8-27B},
  author = {Qwen Team, Alibaba Group},
  year   = {2026}
}