Qwen3.5-9B-MTP-8bit

Qwen3.5-9B-MTP-8bit

This repository contains 8-bit quantized Multi-Token Prediction (MTP) drafter weights for Qwen/Qwen3.5-9B, for use with mlx-vlm speculative decoding. It was quantized from mlx-community/Qwen3.5-9B-MTP-bf16.

This is not a standalone chat or text-generation model. Load it as the draft model alongside a compatible Qwen3.5 9B target checkpoint.

Compatible target models

Pair this drafter with a Qwen3.5-9B target checkpoint. Recommended MLX targets (match target precision to your memory budget):

• mlx-community/Qwen3.5-9B-8bit — recommended pairing for this 8-bit drafter
• mlx-community/Qwen3.5-9B-6bit

Use with mlx-vlm

uv run mlx_vlm.generate \
  --model mlx-community/Qwen3.5-9B-8bit \
  --draft-model ulises-c/Qwen3.5-9B-MTP-8bit \
  --prompt "Hi, how are you?" \
  --max-tokens 256 \
  --enable-thinking

For local weights:

uv run mlx_vlm.generate \
  --model /path/to/target-model \
  --draft-model /path/to/Qwen3.5-9B-MTP-8bit \
  --prompt "Hi, how are you?" \
  --max-tokens 256 \
  --enable-thinking

Model Details

• Model type: qwen3_5_mtp
• MTP block size: 2
• Target architecture: Qwen3.5 9B
• Precision: 8-bit (affine, group size 64) — 8.501 bits per weight
• Runtime: MLX / mlx-vlm
• Format: Safetensors with MLX-compatible config and tokenizer files

Conversion

Quantized with mlx-vlm (the qwen3_5_mtp architecture is provided by mlx-vlm, not mlx-lm):

uvx --from 'mlx-vlm @ git+https://github.com/Blaizzy/mlx-vlm' mlx_vlm.convert \
  --hf-path mlx-community/Qwen3.5-9B-MTP-bf16 \
  --mlx-path Qwen3.5-9B-MTP-8bit \
  -q --q-bits 8 --q-group-size 64

Intended Use

Use this repo only as a speculative decoding drafter for compatible Qwen3.5 9B checkpoints. The target model verifies drafted tokens, while this MTP model proposes candidate tokens per decoding step.

Limitations

This checkpoint requires runtime support for Qwen/DeepSeek MTP draft models in mlx-vlm. Standard standalone generation through generic Transformers APIs is not expected to work with this repository by itself.

Please refer to the upstream Qwen/Qwen3.5-9B model card and license terms for model usage constraints.