Qwen3.8-27B-tq4-g64

Qwen3.8-27B — TurboQuant 4-bit (MLX)

Data-free 4-bit quantization of Qwen/Qwen3.8-27B for Apple Silicon via TurboQuant-MLX.

Qwen3.8-27B is a dense 27.8B multimodal model with a hybrid attention stack: 64 decoder layers, of which 48 are Gated DeltaNet (linear attention) and 16 are full attention — one full layer in every four. It carries a 248,320-token vocabulary and a 27-block vision tower.

• 15.15 GiB on disk, down from 55.6 GB at bf16 — a 3.7× reduction.
• ~22.5 GB peak at 16K context. Comfortable on a 36 GB Mac; runs on 24 GB after raising the Metal wired cap.
• Data-free: no calibration set, no dataset dependency. Randomized Hadamard rotation → per-group RMS scale → Lloyd-Max codebook (optimal for the rotated Gaussian), decoded inline by fused Metal kernels.
• Agentic: passes the Opencode observe → read → edit → verify loop, and does it on a 24 GB Mac with --prefill-step-size 256 (17.63 GiB peak at a 7.2K-token agent prompt). This is the build to pick for agentic use at that size.
• Vision intact: 4/4 on the same battery used for Muse Glimmer.
• Cheap KV: only 16 of 64 layers cache anything, so 64 KB/token — 1.07 GB at 16K.

Read the speed note before you download. This build is compressed and capable, not fast: ~11 tok/s decode, against 29.7 for MLX's own affine 4-bit. See Speed and how this compares.

Documentation: manjunathshiva.github.io/turboquant-mlx: the Qwen3.8-27B page, getting started, choosing a model for your Mac's memory, guides for serving an OpenAI-compatible API, local coding agents, vision models and memory and macOS settings.

Will it fit my Mac?

Real output from the shipped planner (turboquant-plan) for this model:

TurboQuant plan — Qwen3.8-27B-tq4-g64

Model
  type                 qwen3_5  (turboquant 4-bit g64)
  weights (exact)      16.27 GB

Machine
  Metal working set    55.66 GB   ← the real ceiling
  system RAM           68.72 GB

Projection at 16,384 tokens of context
  weights              16.27 GB
  KV cache             1.07 GB  (64.0 KB/token, hybrid: 16/64 full-attention)
  prefill workspace    4.11 GB  (estimate, at --prefill-step-size 2048)
  runtime reserve      1.00 GB  (buffer cache, activations, fragmentation)
                       ----------------------------------
  peak                 22.45 GB of 61.85 GB usable   39.40 GB headroom

Verdict: ✅ RESIDENT — fits fully in memory

Run it against your own machine with turboquant-plan --model <this-repo>. Summary:

How to run

pip install "turboquant-mlx-full[vlm]>=0.25.0"

# text
python -m turboquant_mlx.generate_vlm --model <this-repo> \
    --prompt "Explain Gated DeltaNet in three sentences." --max-tokens 256

# vision
python -m turboquant_mlx.generate_vlm --model <this-repo> \
    --image chart.png --prompt "Which bar is tallest?" --max-tokens 128

# OpenAI-compatible server (vision + tools)
turboquant-serve-vlm --model <this-repo> --port 8080

# tight on memory? cap the prefill chunk (generate and server both take it)
python -m turboquant_mlx.generate_vlm --model <this-repo> \
    --prompt "..." --prefill-step-size 256
turboquant-serve-vlm --model <this-repo> --port 8080 --prefill-step-size 256

<a id="prefill-step-size"></a>

--prefill-step-size is the memory knob for long prompts, and a large one. mlx-vlm only chunks prefill above the step size, so a 1342-token prompt runs unchunked at the 2048 default:

−4.5 GB for 2.1× slower prefill. 256 also keeps every TurboQuant layer on the fused Metal kernel, which materializes nothing.

Vision

Four synthetic images with ground truth drawn into them, so scoring is objective rather than a judgement call — the same battery used on Muse Glimmer, and directly comparable to it.

4/4, peak 18.52 GiB, ~6.6 s per case. Free-form description also produces correct grounded bounding boxes — asked to list every shape in a three-shape image, it returned all three with correct colours and correct positions.

Agentic — and what it needs in memory

Opencode against turboquant-serve-vlm, on the standard task: a tiny repo with a planted off-by-one in average() (len(values) + 1) and a failing pytest suite. Pass = ran the exact given pytest command, read stats.py, made the correct minimal edit, re-ran to green, explained the bug with file:line, and left the tests byte-identical.

Native tool calls are clean (finish_reason: tool_calls, well-formed arguments).

On 24 GB, serve with --prefill-step-size 256. It costs about half the prefill rate (172 → 63 tok/s) and saves 5.1 GiB of transient workspace, which is the difference between fitting a 24 GB machine and not. On 32 GB or more, leave it at the default.

Peaks are measured at a 7,211-token prompt — the size of a real agent's system prompt, which is what sets the floor, not a short chat turn.

This is the build to use for agentic work at 24 GB. The 3-bit tq3-mini-g64 also passes (14.04 GiB at step 256) and is the choice if you need more headroom, but this one is a full perplexity tier better (7.2496 vs 8.0052). Neither build drives an agent on 16 GB — that was tested and failed 0 for 4; see the 3-bit card.

Do not enable prefix caching

mlx-vlm's APC (APC_ENABLED=1) is a NO-GO on this architecture: measured cached_tokens=0 on every request, then a fatal OOM in _clone_prompt_cache_for_apc. APC's self-check reports 48 ArraysCache layers (cap=checkpoint) against 16 KVCache layers (cap=pageable) — Gated-DeltaNet recurrent state cannot be sliced into reusable blocks, so the cache snapshots all 48 layers whole. Leave it off (the default).

Serve with the model's own sampling defaults rather than setting temperature — mlx-vlm merges generation_config.json (temperature 1.0 / top_p 0.95 / top_k 20) and applies it to any field the client omits.

Speed — the honest part

For comparison, Laguna-XS.2 at 3-bit finishes the same Opencode task in 33 s. This model passes the task; it does not feel fast doing it. The cause is the usual codebook-versus-affine trade plus an online Hadamard rotation per layer — the same reason TurboQuant builds decode slower than MLX's affine quantizer at matched bit width.

Download this for the size and the capability, not for latency. If you want a fast local coder, take a smaller model or an affine build.

Worth knowing: 4-bit is both faster and better than 3-bit here. A tq3-g64 build is 12.88 GiB and 9.2 tok/s — smaller, but slower and lower quality. TurboQuant's codebook path only earns its keep at 2-bit; at 3-bit the 10-indices-per-uint32 packing costs more to unpack than 4-bit's clean 8. That is why only the 4-bit build is published.

How this compares to mlx-community's 4-bit

mlx-community/Qwen3.8-27B-4bit is the same architecture at the same 4-bit / group-size-64 setting, with no MTP weights on either side. I measured both through the same harnesses and the same server rather than quoting published numbers:

If you want 4-bit Qwen3.8-27B, take the affine build. It decodes 2.6× faster at a lower peak, and the perplexity difference is 0.3% — stable in ordering across both corpus sizes, but nowhere near worth trading that speed for. This build is worth having when you want the smallest honest 4-bit footprint (--extras-bits 4 puts it at 13.81 GiB, a full 1.14 GiB under the affine build) or you are already running TurboQuant for other models.

The one place the method is structurally ahead is storage. Same 17408×5120 projection, same bit width and group size, in both checkpoints:

The Hadamard rotation symmetrizes each group, so there is no zero-point to store. That is a rounding error at 4-bit, and precisely why the codebook only earns its keep at 2-bit and below.

Two cautions on reading that table, because both invite the wrong conclusion:

• The vision split is not a capability gap. The affine build's single miss is one string (VOLTAGE 47 → VOLTAGE) — a genuine generation, not a scoring artifact, but a follow-up five-string word+number probe put it at 4/5. Treat 3/4-vs-4/4 as a tie.
• The agentic wall-clock is not a speed metric. The affine build decodes 2.6× faster and still finished slower, because it spent more tool turns.

Reproduce either half. The harnesses live in the GitHub repo — they are not part of the pip package — and the perplexity one reads WikiText-2, so it needs the eval extra on top of vlm:

pip install "turboquant-mlx-full[vlm,eval]>=0.25.0"

git clone https://github.com/manjunathshiva/turboquant-mlx
cd turboquant-mlx

python benchmarks/eval_vlm_perplexity.py \
    <this-repo> mlx-community/Qwen3.8-27B-4bit --chunks 64
python benchmarks/eval_vlm_vision.py \
    <this-repo> mlx-community/Qwen3.8-27B-4bit

One implementation note: lm_head is not polar-quantized

With a 248,320-token vocabulary, lm_head is 248320×5120 = 1.271B parameters, the largest matrix in the model. TurboQuant's fused Metal kernel covers batches up to 256 tokens; above that it dequantizes the weight through full-size intermediates. Measured on exactly that shape at 3-bit:

So lm_head is left to an 8-bit affine layer instead, which has a fused batched matmul and materializes no weight at all. End to end on a 1342-token prompt that is 22.85 GB peak versus 33.34 GB, for 0.63 GB more on disk.

This is easy to miss: mlx-vlm's chunked prefill discards the chunk forward's return value, so MLX never evaluates the matmul. But chunking only engages above prefill_step_size (default 2048) — a prompt of 257–2048 tokens takes the single-shot path, which does evaluate it. That is an ordinary chat turn. Fixed in turboquant-mlx#95, released in 0.24.0.

Method

randomized Hadamard rotation   (Gaussianize the weight distribution)
  → group along the input dim  (group size 64)
  → per-group RMS scale
  → clip
  → nearest Lloyd-Max centroid (optimal for N(0,1))
  → 4-bit pack (8 indices per uint32)

Embeddings, the vision tower and lm_head are 8-bit affine (--quantize-extras); everything else is 4-bit TurboQuant. No calibration data is used at any point.

Reproduce:

python -m turboquant_mlx.convert_vlm \
  --hf-path Qwen/Qwen3.8-27B \
  --mlx-path ./Qwen3.8-27B-tq4-g64 \
  --bits 4 --group-size 64 --quantize-extras --extras-bits 8

Conversion takes about 18 seconds on an M4 Max.

Measured on

Apple M4 Max, 64 GB, macOS 26.5.2 · turboquant-mlx 0.24.0 · mlx-lm 0.31.3

Verified on both mlx-vlm 0.6.3 and 0.6.13 — identical memory peaks and identical vision results on each, so the [vlm] floor of 0.6.12 reproduces these numbers.

License

Apache-2.0, inherited from Qwen/Qwen3.8-27B. The upstream LICENSE ships in this repo unchanged.

Citation

@software{turboquant_mlx,
  title  = {TurboQuant-MLX: Data-free weight quantization for MLX on Apple Silicon},
  author = {Manjunath Janardhan},
  year   = {2026},
  url    = {https://github.com/manjunathshiva/turboquant-mlx}
}