Qwythos-9B-v2-MLX-bf16-mtp

Qwythos-9B-v2-MLX-bf16-mtp

This repository, xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp, contains an oMLX-specific bfloat16 conversion of empero-ai/Qwythos-9B-v2 with the model's 15 native Multi-Token Prediction (MTP) tensors embedded under the language_model.mtp.* namespace.

The embedded MTP head is intended for oMLX's Lightning MTP runtime path. It can also be retained when this BF16 model is converted to an oQ model by enabling Preserve MTP weights in oMLX.

Compatibility notice: this embedded-MTP variant is supported only by oMLX. Do not use this repository as an ordinary MLX-VLM, MLX-LM, or LM Studio model. Those runtimes do not provide the oMLX embedded-head loading, quantization, and Lightning MTP behavior required by this artifact.

No additional fine-tuning was performed for this repository. The weights were converted from the upstream v2 checkpoint while preserving BF16 precision, the upstream Apache-2.0 license, and the native MTP head.

Model Variants

This model card covers only the second variant. Its examples intentionally use oMLX exclusively.

Model Summary

• Repository: xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp
• Required runtime: oMLX
• MTP runtime mode: Lightning MTP
• Format: MLX safetensors
• Precision: bfloat16
• Parameters: about 9B plus the native MTP head
• Architecture: Qwen3.5 hybrid vision-language model
• Text layers: 32
• Vision layers: 27
• MTP layers: 1
• Dedicated MTP embeddings: false
• Configured context length: 1,048,576 tokens with YaRN factor 4
• Original YaRN window: 262,144 tokens
• Total tensor count: 775
• Embedded MTP tensor count: 15
• Indexed weight data: 19,306,208,736 bytes
• Output shards: 4
• Maximum shard tensor data: 5 GiB
• Upstream model: empero-ai/Qwythos-9B-v2
• Upstream revision converted: 2178f73a9b5ea28ccd8f6096ef6bac5cd59c9d8b

The model retains the Qwen3.5 visual tower and image/video special tokens. The upstream card primarily reports text reasoning evaluations, so visual performance should be evaluated on the intended workloads.

About the Upstream Model

Qwythos-9B-v2 is an Empero AI reasoning model built on Qwen3.5-9B. The v2 release applies FTPO (Final-Token Preference Optimization) to the earlier Qwythos checkpoint. Instead of broadly modifying the output distribution, FTPO targets token positions that begin repetition loops and trains the model to prefer coherent alternatives.

According to the upstream model card, v2:

• eliminates the repetition and degeneration observed under greedy or low-temperature decoding;
• preserves the reasoning capability of the earlier Qwythos release;
• restores the native Qwen3.5 MTP head;
• removes unsolicited identity preambles;
• retains its one-million-token YaRN configuration and multimodal-capable Qwen3.5 architecture;
• remains intentionally less refusal-oriented for research, cybersecurity, red-teaming, biology, chemistry, pharmacology, and clinical topics.

The upstream developers describe v2 as a hygiene and robustness release rather than a new capability tier. They also note that the restored MTP head came from the Qwen3.5-9B base and was not co-trained with the FTPO-updated main weights. Lightning MTP acceptance may therefore be workload-dependent.

Upstream-Reported Evaluation Results

These results are summarized from the upstream card. They were measured with Empero AI's internal generative chain-of-thought harness and were not independently reproduced by this conversion.

See the upstream model card for the complete methodology, training description, comparisons, examples, and limitations.

Install oMLX

Install oMLX with Homebrew:

brew tap jundot/omlx https://github.com/jundot/omlx
brew install omlx

Start the managed background server:

omlx start

Or run oMLX in the foreground with an explicit model directory:

omlx serve --model-dir /path/to/models

Place the complete Qwythos-9B-v2-MLX-bf16-mtp directory under the selected oMLX model directory. The server automatically discovers model subdirectories. The default service address is:

http://localhost:8000

The instructions and Preserve MTP behavior in this card were verified against oMLX 0.5.1. Use an oMLX release that exposes both Lightning MTP and Preserve MTP weights.

Enable Embedded Lightning MTP

After oMLX discovers the model:

1. Open the oMLX Admin UI at http://localhost:8000/admin.
2. Open Models.
3. Select Qwythos-9B-v2-MLX-bf16-mtp and open its model settings.
4. Confirm that oMLX reports the model as MTP-compatible.
5. Enable Lightning MTP.
6. Save the model settings.

Use Lightning MTP, which consumes the head embedded in this repository. Do not configure the separate VLM MTP external-drafter mode for this embedded variant.

If the Lightning MTP toggle is unavailable, verify that:

• oMLX is a version with native Qwen3.5 MTP support;
• config.json still declares mtp_num_hidden_layers: 1;
• the safetensors index still contains language_model.mtp.* keys;
• conflicting speculative features shown by the oMLX interface are disabled.

Chat in oMLX

The built-in oMLX chat interface is available at:

http://localhost:8000/admin/chat

Select the model ID shown by oMLX and start a text or multimodal conversation. The embedded MTP head is used when Lightning MTP is enabled in the model settings.

OpenAI-Compatible oMLX API

After the oMLX server has discovered the model and Lightning MTP has been enabled, call oMLX's OpenAI-compatible endpoint:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  --data '{
    "model": "Qwythos-9B-v2-MLX-bf16-mtp",
    "messages": [
      {
        "role": "user",
        "content": "Explain speculative decoding with a native MTP head."
      }
    ],
    "temperature": 0.6,
    "top_p": 0.95,
    "max_tokens": 2048
  }'

Use the exact model ID displayed in the oMLX Models page if it differs from the directory name.

Recommended Sampling

The upstream model is designed to reason before answering. A starting point is:

{
  "temperature": 0.6,
  "top_p": 0.95,
  "top_k": 20,
  "repetition_penalty": 1.05,
  "max_tokens": 16384
}

The upstream v2 card reports that repetition penalty is optional rather than load-bearing because the looping behavior was trained out. Use a smaller generation budget for ordinary prompts and a larger budget for difficult reasoning, coding, tool-use, or long-context tasks.

oQ Quantization with Preserve MTP Weights

This BF16 embedded-MTP model is a valid source model for oMLX oQ quantization. To retain native Lightning MTP after quantization, the Preserve MTP weights option must be enabled explicitly.

Recommended Admin UI Procedure

1. Start oMLX and open http://localhost:8000/admin.
2. Open Models and locate oQ Quantization.
3. Select Qwythos-9B-v2-MLX-bf16-mtp as the source model.
4. Select the desired oQ level.
5. Open Advanced Settings.
6. Leave Text Only disabled if the visual tower must remain in the output.
7. Enable Preserve MTP weights.
8. Select the non-quantized weight dtype. bfloat16 is the conservative choice for numerical stability.
9. Start quantization and monitor the task until it completes.
10. Open the generated model's settings and confirm that Lightning MTP remains available.

oMLX 0.5.1 accepts the following oQ levels:

oQ2, oQ2.5, oQ2.7, oQ3, oQ3.5, oQ4, oQ5, oQ6, oQ8

For example, selecting oQ6 with Preserve MTP enabled produces an output name ending in:

-oQ6-mtp

The final name is assigned by oMLX after removing previous precision or quantization suffixes from the source directory name.

What Preserve MTP Weights Does

With Preserve MTP weights enabled, oMLX:

• detects the embedded language_model.mtp.* tensors;
• keeps the MTP tensors in the output shards;
• keeps MTP-related configuration fields in config.json;
• applies its patched Qwen3.5 sanitize and MTP normalization handling;
• keeps the Qwen3.5 MTP fusion projection in full precision to avoid collapsing draft-token acceptance;
• quantizes eligible internal MTP decoder weights using the same sensitivity-driven oQ plan used for compatible backbone layers;
• keeps normalization parameters and other non-quantizable tensors in the selected non-quantized dtype;
• appends -mtp to the output model name;
• leaves the generated model eligible for oMLX's Lightning MTP toggle.

In this context, “Preserve MTP weights” means preserving a complete, loadable, self-consistent MTP head. It does not mean that every eligible matrix inside the MTP decoder necessarily remains BF16.

What Happens If It Is Disabled

Preserve MTP is disabled by default. If it remains off, oMLX:

• removes mtp.* tensors from the output;
• excludes MTP layers from the quantization plan;
• normalizes mtp_num_hidden_layers or equivalent config fields to 0;
• produces a self-consistent non-MTP oQ model;
• does not expose Lightning MTP for the resulting model.

An oQ model created without Preserve MTP cannot regain the original MTP head by changing config.json alone. Re-run quantization from this BF16 source with the option enabled.

oMLX Admin API Payload

The oMLX dashboard submits the following fields to its authenticated POST /admin/api/oq/start endpoint. This example is provided for users who already automate the oMLX Admin API:

{
  "model_path": "/path/to/Qwythos-9B-v2-MLX-bf16-mtp",
  "oq_level": 6,
  "group_size": 64,
  "sensitivity_model_path": "",
  "text_only": false,
  "dtype": "bfloat16",
  "preserve_mtp": true
}

The Admin API requires an authenticated oMLX admin session. For normal use, the oMLX dashboard is the recommended interface.

Before Starting oQ

This rebuilt BF16 source already contains the corrected Qwen3.5 vision patch-embedding layout:

vision_tower.patch_embed.proj.weight
[1152, 2, 16, 16, 3]

The old channels-first layout caused model loading and sensitivity measurement to fail before quantization. The current artifact was independently verified against the source transpose and is the correct source for a new oQ task.

Sensitivity measurement still requires oMLX to load a compatible model or sensitivity proxy. If oQ reports that sensitivity measurement produced no scores, inspect the earlier log lines for the actual model-load or calibration failure rather than treating that final message as the root cause.

After Quantization

In the oMLX Models page, verify all of the following:

• the output model name ends in -mtp;
• oMLX reports the output as MTP-compatible;
• Lightning MTP can be enabled;
• the model loads without missing or unexpected MTP parameter errors;
• a short chat request completes with Lightning MTP enabled;
• the oMLX logs report MTP activity rather than falling back because the weights are absent.

Measure target-only and Lightning MTP throughput with the same prompt, sampling parameters, output-token count, and concurrency. The upstream card warns that the restored MTP head was not co-trained with the FTPO update, so acceptance and speedup can vary.

Conversion Notes

Conversion Provenance

• Source: empero-ai/Qwythos-9B-v2
• Source revision: 2178f73a9b5ea28ccd8f6096ef6bac5cd59c9d8b
• Source weights: one 19,306,305,296-byte safetensors file
• Output dtype: BF16
• Output tensor count: 775
• MTP tensor count: 15
• Output weight data: 19,306,208,736 bytes
• Output shards: 4
• Maximum shard tensor data: 5 GiB
• Additional training: none

The source checkpoint was converted with a streaming, GPU-independent safetensors converter. Unchanged BF16 tensors were copied without a full-model BF16-to-FP32 round trip. Only framework-required tensor transformations were applied.

Tensor Transformations

1. Hugging Face tensor names were remapped to the MLX Qwen3.5 namespace.
2. The 15 source mtp.* tensors were retained as language_model.mtp.*.
3. Qwen3.5 RMSNorm weights received the MLX-required +1 offset.
4. Linear-attention Conv1d weights were moved from [out, 1, kernel] to [out, kernel, 1].
5. The visual patch-embedding Conv3d weight was transposed from [1152, 3, 2, 16, 16] to MLX channels-last layout [1152, 2, 16, 16, 3].
6. The single source weight file was automatically divided into four shards with no more than 5 GiB of tensor data per shard.

Compatibility Fixes Applied

partial_rotary_factor

The upstream config stores partial_rotary_factor at the text_config top level, while the relevant MLX Qwen3.5 paths consume it from rope_parameters. The converted config stores:

{
  "text_config": {
    "rope_parameters": {
      "rope_type": "yarn",
      "factor": 4.0,
      "original_max_position_embeddings": 262144,
      "mrope_section": [11, 11, 10],
      "rope_theta": 10000000,
      "partial_rotary_factor": 0.25
    }
  }
}

vision_config.model_type

The upstream value qwen3_5_vision was normalized to qwen3_5 for compatibility with MLX backends that use a narrower vision model-type whitelist.

Vision Conv3d Weight Layout

The visual patch projection was changed from Hugging Face channels-first Conv3d layout:

[out, in, depth, height, width] = [1152, 3, 2, 16, 16]

to MLX channels-last kernel layout:

[out, depth, height, width, in] = [1152, 2, 16, 16, 3]

Both the safetensors metadata and the underlying BF16 data were transposed.

Verification

The conversion was independently checked:

• all 775 tensors are unique and BF16;
• exactly 15 tensors belong to language_model.mtp.*;
• the safetensors index exactly matches the shard contents;
• each shard contains at most 5 GiB of tensor data;
• config.json declares one MTP hidden layer;
• partial_rotary_factor is present in rope_parameters;
• vision_config.model_type is qwen3_5;
• every BF16 element of the converted visual patch projection matches the expected source transpose.

oMLX's compatibility detector also checks both the MTP config fields and the actual presence of mtp.* tensors. This repository satisfies both conditions.

Files

Function Calling

The upstream Qwythos model uses the Qwen3.5 chat template and is intended for tool-augmented and agentic workflows. Use oMLX's OpenAI-compatible API and handle tool execution, argument validation, permissions, and safety policy in the calling application.

Limitations and Safety

• This embedded-MTP variant is supported only by oMLX.
• It is not the standalone --draft-model artifact; the MTP head is embedded in the main model shards.
• Lightning MTP must be enabled in oMLX model settings to use the embedded head during generation.
• Preserve MTP weights must be enabled explicitly for any oQ conversion that should retain Lightning MTP.
• Speculative acceptance and speedup depend on workload and hardware. No fixed speedup is claimed.
• This is a format conversion, not a new training run. It does not improve or independently validate the upstream model's capabilities.
• The benchmark values above are upstream-reported internal-harness results.
• The model may hallucinate facts, citations, identifiers, code behavior, or tool arguments. Use retrieval, execution checks, and human review when correctness matters.
• The upstream release is intentionally less refusal-oriented and may engage with sensitive medical, biological, chemical, or security topics. Add application-level safeguards and comply with applicable law.
• Do not use the model as the sole authority for medical, legal, financial, security-critical, or other high-stakes decisions.
• The configured one-million-token context can require substantial unified memory and has not been validated for every modality or deployment.
• Visual-language quality was not independently benchmarked as part of this conversion.

License

This oMLX-specific conversion is released under the same Apache-2.0 license as the upstream model.

Acknowledgements

• Original model: empero-ai/Qwythos-9B-v2
• Original developer: Empero AI
• Base family: Qwen3.5-9B by Alibaba/Qwen
• Required runtime: oMLX
• oMLX conversion repository: xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp