Qwythos-9B-v2-MLX-oQ8-mtp

Qwythos-9B-v2-MLX-oQ8-mtp

This repository, xunkutech-ai/Qwythos-9B-v2-MLX-oQ8-mtp, contains an oMLX-specific oQ8 quantization of xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp. It uses 8-bit affine quantization with group size 64 and retains the model's native Multi-Token Prediction (MTP) head for oMLX Lightning MTP.

No additional fine-tuning was performed. This artifact changes storage and inference precision only.

Compatibility notice: this embedded-MTP model is intended for oMLX. Standard MLX-VLM, MLX-LM, and LM Studio runtimes do not provide the embedded MTP loading, quantization, and Lightning MTP behavior required by this artifact. A local directory under ~/.lmstudio/models does not by itself make the model compatible with LM Studio's standard MLX runtime.

Model Variants

Model Summary

• Repository: xunkutech-ai/Qwythos-9B-v2-MLX-oQ8-mtp
• Required runtime: oMLX
• MTP runtime mode: Lightning MTP
• Format: MLX safetensors
• Source precision: BF16
• Quantization: affine oQ8, group size 64
• Parameters: about 9B plus the native MTP head
• Architecture: Qwen3.5 hybrid vision-language model
• Text layers: 32
• Vision layers: 27
• MTP layers: 1
• Configured context length: 1,048,576 tokens with YaRN factor 4
• Original YaRN window: 262,144 tokens
• Indexed tensor entries: 1,289, including quantization scales and biases
• Indexed MTP-related entries: 29
• Indexed weight data: 10,716,567,097 bytes
• Output shards: 3
• Quantization source: xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp
• Original upstream model: empero-ai/Qwythos-9B-v2

The visual tower, image special tokens, one-million-token YaRN configuration, Qwen3.5 chat template, and embedded MTP head are retained from the BF16 source.

About the Upstream Model

Qwythos-9B-v2 is an Empero AI reasoning model built on Qwen3.5-9B. The v2 release applies Final-Token Preference Optimization (FTPO) to reduce repetition loops while preserving the reasoning capability of the earlier checkpoint. It also restores the native Qwen3.5 MTP head.

The upstream developers describe v2 as a hygiene and reliability release rather than a new capability tier. The restored MTP head came from the Qwen3.5-9B base and was not co-trained with the FTPO-updated main weights, so Lightning MTP acceptance remains workload-dependent.

Upstream-Reported Evaluation Results

The following values come from the upstream model card and were not independently reproduced during this quantization:

See the upstream model card for its full methodology, training description, and limitations.

Run with oMLX

Install and start oMLX:

brew tap jundot/omlx https://github.com/jundot/omlx
brew install omlx
omlx start

Alternatively, point oMLX at an explicit model directory:

omlx serve --model-dir /path/to/models

Place the complete Qwythos-9B-v2-MLX-oQ8-mtp directory under the selected oMLX model directory. Then open the oMLX Admin UI, select the model, and enable Lightning MTP. Use the embedded Lightning MTP mode, not a separate external draft model.

If the Lightning MTP option is unavailable, verify that:

• the oMLX version supports native Qwen3.5 MTP;
• config.json still declares mtp_num_hidden_layers: 1;
• the weight index still contains language_model.mtp.* entries;
• no conflicting speculative-decoding mode is enabled.

OpenAI-Compatible API

Use the exact model ID shown in the oMLX Models page if it differs from the directory name.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  --data '{
    "model": "Qwythos-9B-v2-MLX-oQ8-mtp",
    "messages": [
      {
        "role": "user",
        "content": "Explain how native MTP speculative decoding works."
      }
    ],
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "repetition_penalty": 1.05,
    "max_new_tokens": 16384
  }'

Clients that expose the OpenAI-standard max_tokens field instead of max_new_tokens may use that equivalent server-supported field.

Recommended Sampling

The configuration below passed all expanded Hermes tool-calling cases on this oQ8 build:

{
  "temperature": 0.6,
  "top_p": 0.95,
  "top_k": 20,
  "repetition_penalty": 1.05,
  "max_new_tokens": 16384
}

For more varied responses, the following higher-temperature profile also performed well in the Hermes regression suite:

{
  "temperature": 0.7,
  "top_p": 0.95,
  "top_k": 60,
  "min_p": 0
}

The 16,384-token budget is intended for difficult reasoning, coding, long context, or agent tasks. Use a smaller limit for ordinary chat and short tool calls. The upstream model was trained to reduce repetition loops, so the 1.05 repetition penalty is conservative rather than load-bearing.

Quantization Details

The local configuration was inspected after quantization:

• all eligible matrices use 8-bit affine quantization with group size 64;
• one native MTP hidden layer remains enabled;
• the index contains 29 MTP-related entries after quantized weights, scales, and biases are counted;
• eligible MTP decoder matrices follow the oQ plan while required non-quantizable tensors remain available to the runtime;
• the output remains eligible for oMLX Lightning MTP.

Compared with oQ6, oQ8 uses more unified memory but preserves token probabilities more closely and showed the strongest tool-calling stability in the higher-temperature comparison.

Hermes Agent Tool-Calling Evaluation

These tests evaluate operational tool-calling behavior, not general benchmark quality. They were run through a local OpenAI-compatible oMLX endpoint using tool_choice="auto".

Method

Each test used two real model inference turns:

1. The model selected a read or inspection tool.
2. The harness returned a deterministic simulated tool result.
3. The model had to call the required side-effect tool with valid arguments.

The harness covered three workflows:

• read a file and append an item only when it is absent;
• inspect an unhealthy service and restart it;
• inspect a failed deployment and roll it back to the reported stable version.

A run passed only when the model returned the correct tool_calls and critical arguments. A prose claim such as "the file was updated" did not count as success. External side effects were simulated; the test did not modify a real production service.

Fixed-Seed Quantization Comparison

The first phase used the same file workflow, the same 15 seeds, and this configuration:

temperature: 0.6
top_p: 0.95
top_k: 60
min_p: 0

The main oQ4 failure mode was a correct first read followed by prose claiming that the write had completed without an actual write tool call.

Expanded Three-Workflow Comparison

The expanded phase used temperature=0.6, top_p=0.95, top_k=20, repetition_penalty=1.05, and max_new_tokens=16384. It did not set a seed, so it measures repeated operational stability rather than deterministic token-for-token equivalence.

Higher-Temperature oQ6 and oQ8 Comparison

At temperature=0.7, top_p=0.95, top_k=60, and min_p=0, a paired 16-case test produced:

An additional 30-case oQ8 campaign passed 30/30, giving this build a cumulative 46/46 under the higher-temperature configuration.

Interpretation and Recommendation

• oQ4 saves the most memory but showed repeated failures in the second step of multi-turn tool workflows.
• oQ5 was much better than oQ4 in the fixed-seed test and can be a constrained memory compromise.
• oQ6 is the minimum recommended precision for Hermes-style autonomous tool use and passed all expanded cases with the recommended sampling profile.
• oQ8 is the recommended quantized variant when memory permits. It matched oQ6 at the conservative profile and was more consistent in the paired higher-temperature comparison.

No quantization level makes side-effect execution inherently safe. The calling application should require real tool calls, validate arguments, enforce permissions, and retry or escalate when a required action is missing.

Verification

The local oQ8 artifact was checked for the following properties:

• config.json declares 8-bit affine quantization with group size 64;
• mtp_num_hidden_layers is 1;
• the index contains 1,289 entries and exactly 29 MTP-related entries;
• the index references exactly three weight shards;
• indexed tensor data totals 10,716,567,097 bytes;
• the YaRN factor, original context window, partial rotary factor, and Qwen3.5 vision model type remain present.

Files

Limitations and Safety

• This embedded-MTP artifact is intended for oMLX and is not a standalone --draft-model repository.
• Lightning MTP must be enabled in oMLX to use the embedded head.
• Quantization can change token probabilities and multi-turn tool behavior even when conventional answer quality appears similar.
• The Hermes evaluation is a local regression test, not a standardized or statistically exhaustive benchmark. Results can vary with oMLX version, hardware, concurrency, chat template, and sampling implementation.
• The tests cover short tool calls. They do not validate 16,384-token output, one-million-token context accuracy, throughput, latency, or visual quality.
• The model may hallucinate facts, citations, code behavior, or tool arguments. Validate all side-effecting actions in the calling application.
• The upstream release is intentionally less refusal-oriented. Add safeguards appropriate to medical, biological, chemical, security, and other sensitive use cases.
• Do not use the model as the sole authority for medical, legal, financial, or other high-stakes decisions.

License

This quantized conversion is released under the same Apache-2.0 license as the upstream model.

Acknowledgements

• Original model: empero-ai/Qwythos-9B-v2
• BF16 oMLX source: xunkutech-ai/Qwythos-9B-v2-MLX-bf16-mtp
• Original developer: Empero AI
• Base family: Qwen3.5-9B by Alibaba/Qwen
• Required runtime: oMLX