Qwythos-27B-v1-MLX-VLM-4bit

Qwythos-27B-v1 · MLX VLM 4-bit

Community MLX conversion of empero-ai/Qwythos-27B-v1 for Apple Silicon. Vision tower kept. MTP head is not inside this repo — use the companion drafter.

Start here if you have ~32GB unified memory. This is the everyday chat / vision build.

Which repo should I download?

Official GGUF (LM Studio / llama.cpp), including baked-in MTP: empero-ai/Qwythos-27B-v1-GGUF.

Install

pip install -U mlx-vlm

Apple Silicon only. Needs mlx-vlm >= 0.6.

Quick start

Text

mlx_vlm.generate \
  --model FancieF/Qwythos-27B-v1-MLX-VLM-4bit \
  --prompt "Introduce yourself in one sentence." \
  --thinking-mode disabled \
  --max-tokens 256 \
  --temperature 0.6

Vision

mlx_vlm.generate \
  --model FancieF/Qwythos-27B-v1-MLX-VLM-4bit \
  --image ./photo.jpg \
  --prompt "Describe this image." \
  --thinking-mode disabled \
  --max-tokens 256 \
  --temperature 0.6

Vision + MTP (image prefill on the main model; MTP speeds text decode)

mlx_vlm.generate \
  --model FancieF/Qwythos-27B-v1-MLX-VLM-4bit \
  --draft-model FancieF/Qwythos-27B-v1-MLX-MTP \
  --draft-kind mtp \
  --draft-block-size 4 \
  --image ./photo.jpg \
  --prompt "Describe this image." \
  --thinking-mode disabled \
  --temperature 0.6

Local path works the same after hf download / ModelScope snapshot.

LM Studio note

• This MLX 4-bit + MLX MTP pair is meant for mlx-vlm CLI/server, not LM Studio’s “Draft Model” picker (native MTP head ≠ a small peer LLM).
• For LM Studio MTP, prefer official Qwythos-27B-MTP-Q4_K_M.gguf and enable MTP Speculative Decoding.

What’s inside

• Affine 4-bit MLX safetensors (group_size=64)
• language_model + vision_tower (333 vision tensors)
• Chat template / processor configs from upstream
• No mtp.* weights (stripped by mlx_vlm.convert; recovered in the MTP repo)

Sampling

From upstream / Qwen3.5 thinking defaults:

Rough M5 32GB (short prompts, thinking off): ~6 tok/s alone · ~8–11 tok/s with MTP.

Conversion

mlx_vlm.convert \
  --hf-path /path/to/Qwythos-27B-v1 \
  --mlx-path ./Qwythos-27B-MLX-VLM-4bit \
  -q --q-bits 4 --dtype bfloat16 --trust-remote-code

Use mlx_vlm.convert, not mlx_lm.convert, if you need vision.

License & attribution

• Derived from empero-ai/Qwythos-27B-v1 (Apache-2.0), based on Qwen3.5-27B.
• Unofficial community MLX quant for Apple Silicon inference.
• Upstream is uncensored for technical/research use — add your own app-level controls.

Links