Qwythos-9B-v2-MLX-bf16-mtp-draft

Qwythos-9B-v2-MLX-bf16-mtp-draft

This repository, xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp-draft, contains the standalone native Qwen3.5 Multi-Token Prediction (MTP) drafter for:

xunkutech-ai/Qwythos-9B-v2-MLX-bf16

The 15 BF16 MTP tensors were extracted from empero-ai/Qwythos-9B-v2, sanitized for MLX-VLM's qwen3_5_mtp drafter implementation, and saved separately so MLX-VLM can load them through the --draft-model interface.

Important: this repository is an auxiliary drafter, not a standalone language or vision-language model. Do not pass it as the main --model. Pair it with the matching Qwythos-9B-v2 MLX BF16 target model shown above.

No additional fine-tuning was performed for this repository. The weights remain bfloat16 and inherit the upstream Apache-2.0 license.

Model Pair

The target model verifies every candidate token proposed by this drafter. Speculative decoding changes the decoding path and performance characteristics; it does not allow unverified draft tokens to become final output.

Drafter Summary

• Repository: xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp-draft
• Role: standalone Qwen3.5 native MTP speculative drafter
• Required target: xunkutech-ai/Qwythos-9B-v2-MLX-bf16
• Framework: MLX-VLM
• Format: one MLX safetensors file
• Precision: bfloat16
• Tensor count: 15
• Weight file size: 486,582,819 bytes
• model_type: qwen3_5_mtp
• Configured block_size: 3
• MTP depth: 1 hidden layer
• Dedicated MTP embeddings: false
• Hidden size: 4096
• Vocabulary size: 248,320
• Configured context length: 1,048,576 tokens
• Upstream source: empero-ai/Qwythos-9B-v2
• Upstream revision: 2178f73a9b5ea28ccd8f6096ef6bac5cd59c9d8b

This drafter contains no language-model trunk, LM head, visual tower, image processor, or video processor. It relies on the paired target model for those components.

About the Upstream Model

Qwythos-9B-v2 is an Empero AI reasoning model built on Qwen3.5-9B. According to the upstream model card, v2 applies FTPO (Final-Token Preference Optimization) to eliminate the looping and degeneration found in the earlier Qwythos release while preserving its reasoning behavior, one-million-token YaRN configuration, and multimodal-capable Qwen3.5 architecture.

The upstream v2 checkpoint restores the native MTP head that was missing from the previous export. The upstream developers also note that the restored MTP head came from the Qwen3.5-9B base and was not co-trained with the FTPO-updated main weights. Consequently, speculative acceptance may be modest and should be measured on the intended prompts and hardware.

For benchmark results, sampling guidance, capabilities, limitations, and safety information, see:

• xunkutech-ai/Qwythos-9B-v2-MLX-bf16
• empero-ai/Qwythos-9B-v2

What MTP Does

Multi-Token Prediction adds a shallow auxiliary model that proposes several future tokens from target-model hidden states. The full target model then verifies those candidates. When several candidates are accepted together, fewer full autoregressive target passes may be required.

Actual speedup depends on:

• candidate-token acceptance rate;
• prompt and output distribution;
• requested draft block size;
• target-model size and quantization;
• unified-memory bandwidth and device generation;
• MLX-VLM version and speculative-decoding implementation.

This repository does not claim a fixed speedup figure.

Install

pip install -U mlx-vlm

or:

uv pip install -U mlx-vlm

Use an MLX-VLM release that includes the Qwen3.5 qwen3_5_mtp drafter.

Quick Start

Load the target and drafter by their Hugging Face repository identifiers:

mlx_vlm.generate \
  --model xunkutech-ai/Qwythos-9B-v2-MLX-bf16 \
  --draft-model xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp-draft \
  --draft-kind mtp \
  --draft-block-size 4 \
  --prompt "Explain speculative decoding with MTP." \
  --max-tokens 2048 \
  --temperature 0.6

The --draft-kind mtp argument explicitly selects the native MTP drafter family. The --draft-block-size 4 argument is an optional runtime override. Omit it to use the packaged default block size of 3.

Local Directory Usage

If both repositories are already downloaded:

mlx_vlm.generate \
  --model /path/to/Qwythos-9B-v2-MLX-bf16 \
  --draft-model /path/to/Qwythos-9B-v2-MLX-bf16-mtp-draft \
  --draft-kind mtp \
  --draft-block-size 4 \
  --prompt "Write a Python implementation of merge sort." \
  --max-tokens 2048 \
  --temperature 0.6

Do not merge model.safetensors from this repository into the main model directory. Current MLX-VLM expects the sanitized qwen3_5_mtp weights in a separate drafter directory.

Server Usage

MLX-VLM can also load the pair for server inference:

mlx_vlm.server \
  --model xunkutech-ai/Qwythos-9B-v2-MLX-bf16 \
  --draft-model xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp-draft \
  --draft-kind mtp \
  --draft-block-size 4

If a bundled application backend does not expose --draft-model and --draft-kind mtp, it cannot use this standalone drafter through this interface. The main target model can still be loaded without speculative decoding.

Sampling Guidance

Sampling parameters belong to the target-model generation request, not to the drafter repository. A starting point inherited from the upstream model card is:

generation_kwargs = {
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "repetition_penalty": 1.05,
    "max_tokens": 16384,
}

Use a shorter generation budget for ordinary prompts. For throughput comparisons, keep the prompt, target model, sampling parameters, random seed, and generated-token count identical between drafter-enabled and target-only runs.

Extraction and Conversion

Source

• Source model: empero-ai/Qwythos-9B-v2
• Source revision: 2178f73a9b5ea28ccd8f6096ef6bac5cd59c9d8b
• Source format: one Hugging Face safetensors file
• Source MTP namespace: mtp.*
• Source MTP tensor count: 15

Transformations

1. Only tensors beginning with the source mtp. prefix were selected.
2. The leading mtp. prefix was removed for the standalone drafter namespace expected by qwen3_5_mtp.
3. Qwen3.5 RMSNorm weights received the MLX-required +1 offset.
4. All tensors were written as BF16 in one safetensors file with format=mlx metadata.
5. The source text configuration was copied into the drafter config.
6. partial_rotary_factor was placed inside rope_parameters for MLX-VLM compatibility.
7. Tokenizer files were copied from the same upstream revision.

No visual weights, base-model attention layers, base-model MLP layers, or general-purpose output head were copied into the drafter.

Configuration

The repository uses the standalone drafter structure documented by MLX-VLM:

{
  "model_type": "qwen3_5_mtp",
  "block_size": 3,
  "tie_word_embeddings": false,
  "text_config": {
    "model_type": "qwen3_5_text",
    "hidden_size": 4096,
    "mtp_num_hidden_layers": 1,
    "mtp_use_dedicated_embeddings": false,
    "max_position_embeddings": 1048576,
    "dtype": "bfloat16"
  }
}

The packaged block_size=3 is the drafter's default speculative block setting derived from the one-layer MTP configuration. MLX-VLM permits a runtime override through --draft-block-size.

Weight Contents

The single weight file contains:

• one MTP fusion projection;
• one MTP transformer layer;
• its attention projections and Q/K normalization;
• its MLP projections;
• input and post-attention normalization;
• final MTP normalization;
• embedding-hidden pre-fusion normalization.

The sanitized keys intentionally do not begin with mtp. because the entire repository is already loaded as a qwen3_5_mtp drafter.

Verification

The extracted artifact was independently checked:

• model.safetensors contains exactly 15 tensors;
• all 15 tensors are BF16;
• safetensors metadata is format=mlx;
• no key retains the leading mtp. prefix;
• no vision or patch-embedding tensor is present;
• config.json declares model_type qwen3_5_mtp;
• config.json declares block_size 3 and one MTP hidden layer;
• tokenizer.json and tokenizer_config.json are present;
• partial_rotary_factor is stored in text_config.rope_parameters.

Weight-file SHA-256:

c8625300dec271742b17b70471d7dbd6c9b118a9035e13d53e735b84d1d298b4

Files

Compatibility

This drafter is designed for the matching Qwythos-9B-v2 MLX BF16 target. Using it with a different tokenizer, vocabulary, hidden size, architecture, or fine-tuned target can fail compatibility checks or produce a low candidate acceptance rate.

The drafter requires a runtime with native Qwen3.5 MTP speculative-decoding support. A runtime that merely loads ordinary MLX safetensors is not sufficient.

Limitations and Safety

• This repository cannot generate text or process images by itself.
• It does not contain the main Qwythos language trunk or vision tower.
• It is intended for xunkutech-ai/Qwythos-9B-v2-MLX-bf16; compatibility with other targets is not claimed.
• The upstream MTP weights were restored from the Qwen3.5-9B base and were not co-trained with the FTPO-updated Qwythos-v2 target, so acceptance may be modest.
• No independent speculative throughput benchmark is claimed by this model card.
• Output quality, factuality, tool-use safety, long-context behavior, and content risks are properties of the paired target model. Review the target and upstream model cards before deployment.
• Do not use the paired model as the sole authority for medical, legal, financial, security-critical, or other high-stakes decisions.

License

This MLX drafter is released under the same Apache-2.0 license as the upstream model.

Acknowledgements

• Matching MLX target: xunkutech-ai/Qwythos-9B-v2-MLX-bf16
• Original model: empero-ai/Qwythos-9B-v2
• Original developer: Empero AI
• Base family: Qwen3.5-9B by Alibaba/Qwen
• MLX ecosystem: Apple MLX and MLX-VLM
• MLX drafter repository: xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp-draft