Tess-4-27B-MLX-Q8

<!-- base_model + base_model_relation are what build the HF "Model tree". With relation: quantized this repo appears under migtissera/Tess-4-27B's *Quantizations* tab automatically -- no manual linking needed. The chain further upstream to Qwen/Qwen3.6-27B is inherited from the base repo's own card, so it renders only if Migel declared it there; that part is not ours to control. -->

Tess-4-27B — MLX 8-bit, with a working MTP head

An MLX conversion of migtissera/Tess-4-27B (Qwen3.6-27B base, dense, vision-capable) quantized to 8-bit (group size 64, affine), with the native multi-token-prediction head restored as a BF16 sidecar — which, at the time of upload, no other public MLX conversion of the Qwen3.6 family ships.

A 4-bit build with the same MTP sidecar is at Tess-4-27B-MLX-Q4 (16.1 GB).

Why the MTP sidecar matters

mlx-lm's qwen3_5 sanitize() unconditionally drops mtp.* tensors during conversion, so every community MLX checkpoint of this family carries mtp_num_hidden_layers: 1 in its config while containing zero MTP weights. Check your model.safetensors.index.json, not the config — the config will tell you the head is there when it is not.

This repo re-extracts the head from the original BF16 release and ships it at mtp/weights.safetensors (15 tensors, BF16 — quantizing the head collapses acceptance to ~0%).

Measured with vllm-mlx 0.4.0 (--enable-mtp, greedy envelope) on an M5 Max: 78% draft acceptance, +19.5% decode throughput with acceptance-verified speculative decoding — no output change vs. non-speculative greedy.

An honest caveat. MTP is real and it works, but it is worth less than it sounds. On the same model and workload, omlx decodes at 18.4 tok/s with no speculative decoding at all, versus vllm-mlx's 19.4 tok/s with MTP (and ~17.3 without). A better runtime recovered nearly the whole gain on its own. Take the head because it is free; do not architect around it.

Benchmarks, and what you gain by choosing 8-bit over 4-bit

Same harness, same 3500-token budget, same machine (M5 Max). The Q4 build is the same conversion at 4 bits, so this is a clean read on what the extra 14 GB buys you:

The honest summary: 8-bit buys you ~8 points of coding and ~7 of long-context. It buys you nothing on reasoning, tool-use, browser action-selection or vision — those survive 4-bit intact. And it costs you 40% of your decode speed and 14 GB.

Take Q8 if you lean on long-context recall or code generation, and you have the memory. Take Q4 if your workload is agentic (tool calls, short-to-medium turns, vision), or you are on a 32 GB machine — there it is close to free.

Small subsets: treat as directional, not as leaderboard numbers. The reasoning axis showing Q4 ahead is well within the noise of a 40-question set — read it as "no loss", not "Q4 is smarter".

Usage

# omlx (recommended — faster engine, and the only one I trust with large images)
omlx serve --model-dir <dir-containing-this-model>

# vllm-mlx (if you want the MTP head to actually engage)
vllm-mlx serve <this-repo> --mllm --enable-mtp --mtp-num-draft-tokens 1
# MTP engages only on exactly-greedy requests: temperature=0, top_p=1, top_k=0, min_p=0

Anything that loads standard MLX safetensors will run this model; engines without MTP support simply ignore the mtp/ sidecar.

Known ecosystem traps this conversion accounts for

• Do not load MLX-format Qwen3.6 checkpoints with mlx-vlm 0.6.4 — it re-applies a +1.0 RMSNorm shift to already-converted weights and produces deterministic garbage, with no error and no warning. Use ≥0.6.5 or ≤0.6.3.
• Do not serve large images through vllm-mlx 0.4.0's batched MLLM path — it silently drops image context and answers anyway, confidently and with fabricated detail (0/2 on a hard screenshot test while inventing numbers; omlx scored 2/2). Serve vision without continuous batching, or via omlx.
• Do not benchmark this model with a short generation cap. It is a reasoning model: a 768-token cap does not truncate its answer, it truncates its thinking, so it emits nothing and scores near zero on problems it can solve. Budget ≥3500 tokens.

Provenance & credits

• Base model: migtissera/Tess-4-27B by Migel Tissera — post-trained on long-context agentic traces atop Qwen/Qwen3.6-27B. All model capabilities are theirs; this repo is packaging.
• Conversion: mlx_vlm convert (8-bit, gs64) + BF16 MTP re-attachment.
• License: Apache-2.0, inherited from the base model.

Converted and benchmarked as part of a local inference stack for a personal Automated Agentic Software Factory — something I'm building solo and will make publicly available after its limited-alpha phase. Full benchmark data, harnesses and the conversion recipe: https://huggingface.co/datasets/studioburnside/mlx-local-inference-benchmarks.

Questions, results and corrections all welcome in Discussions.