Qwen3.8-27B-Hybrid-Q8Q4-Vision

Qwen3.8-27B MLX Hybrid Q8/Q4 + Vision

adrianmurray/Qwen3.8-27B-Hybrid-Q8Q4 with the vision tower grafted back on. The language model weights are Adrian Murray's, bit-for-bit — this repo adds the 333 model.visual.* tensors from the official Qwen/Qwen3.8-27B checkpoint and nothing else.

Why this exists

mlx_lm.convert drops every vision tensor unconditionally (mlx_lm/models/qwen3_5.py:387), so any Qwen3.8-27B converted with it is text-only. The language model is intact; the ViT is simply absent, along with vision_config. This repo restores it.

One directory now serves both paths:

Measured (Apple M1 Max, 64 GB)

Text throughput is unchanged from Adrian's original — the extra tensors are ignored by the text path, so the graft is free.

Requirements

mlx-vlm >= 0.6.15. Earlier versions (including 0.6.3) apply a +1.0 norm shift unconditionally in the qwen3_5 sanitizer, which double-shifts weights that are already in post-sanitize form and silently corrupts the model — it still loads and still answers. 0.6.15 added the should_shift_norm_weights guard that mlx_lm 0.31.3 has.

Built and verified with mlx==0.32.0, mlx-lm==0.31.3, mlx-vlm==0.6.15.

How it was built

1. All 333 model.visual.* tensors live in shard 1 of 18 of the base checkpoint (460.7 M params, 0.92 GB bf16), so only ~4 GB had to be fetched, not 54 GB.
2. Renamed to post-sanitize form (model.visual.* → vision_tower.*) to match how Adrian's language weights are already stored (language_model.model.*). mlx_vlm's sanitize_key passes vision_tower. through untouched.
3. Language-model shards copied byte-for-byte from Adrian's repo.
4. config.json = Adrian's, plus the base vision_config, plus language_model_only: false. His per-module quantization dict is carried over verbatim.

The vision tower is deliberately not quantized — it is 460 M params and runs once per image at prefill rather than per generated token, so quantizing it would save ~0.4 GB and no decode time while risking ViT quality. mlx_vlm's class predicate falls back to f"{p}.scales" in weights, so unlisted modules are correctly left alone.

Verification

The graft is checked two ways, because mlx_vlm silently skips mis-keyed vision tensors rather than erroring — a randomly-initialized tower still produces fluent, plausible answers about an image:

1. Unguessable OCR. A synthetic invoice with a random code; the model returned QK7-95KW4M, 14 March 2027, Bay 12, Kestrel Wharf, 1,483.62 — all exact.
2. Tensor audit. 333/333 vision_tower.* tensors live in the loaded model, none all-zero, spot-checked byte-identical to the shard on disk.

A "describe the shapes" test is not sufficient — shape, colour and position are guessable from the prompt alone.

Credit

The quantization — the hard part, and the reason this runs at 22 tok/s — is entirely Adrian Murray's. See adrianmurray/Qwen3.8-27B-Hybrid-Q8Q4 for the scheme, and adrianmurray/Qwen3.8-27B-MTP-MLX-6bit for the matching native-MTP draft model.

Original model: Qwen/Qwen3.8-27B © Alibaba Cloud, Apache 2.0.