Ornith-1.0-35B-MLX-oQ7

Ornith-1.0-35B-oQ7

MLX format quantization of deepreinforce-ai/Ornith-1.0-35B, produced with mlx-optiq. This repository also ships a grafted Multi-Token Prediction (MTP) head (mtp.safetensors) for speculative decoding on Apple Silicon, sourced from mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit. Testing found that this MTP head does not currently work correctly with this oQ7 mixed-precision quantization, which is why this repository was renamed from Ornith-1.0-35B-oQ7-mtp to Ornith-1.0-35B-oQ7. The mtp.safetensors file has not been removed and remains in the repo for future compatibility work — see Multi-Token Prediction below for details.

Model Details

Model Description

This repository contains a mixed precision MLX quantization of Ornith-1.0-35B, a Qwen3.5-35B-A3B based mixture of experts model. The quantization was produced with mlx-optiq, which performs a per-layer KL sensitivity analysis against the original BF16 checkpoint and assigns each layer either 6-bit or 8-bit precision so that the weighted average lands at a target of 7.5 bits per weight (BPW). Layers that are more sensitive to quantization error are kept at 8-bit, while more robust layers are reduced to 6-bit.

An auxiliary MTP head, grafted from mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit, is included as mtp.safetensors. It is architecturally compatible with the base model because Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share the same hidden size, layer count, expert count, and block layout, and Ornith is fine-tuned directly from Qwen3.5-35B-A3B. However, testing concluded that this head does not work correctly as a self-speculative draft model with the oQ7 mixed-precision quantization in this repository, so it should not currently be enabled for inference. The file is kept in the repo for future compatibility work (see the Multi-Token Prediction section below).

• Developed by: programmer-666
• Model type: Causal decoder only mixture of experts language model (Qwen3.5-35B-A3B architecture, qwen35moe, 40 layers)
• Language(s): Inherited from the base model
• License: Apache 2.0 for this repository. See the Licenses section below for the licenses of the underlying components.
• Quantized from model: deepreinforce-ai/Ornith-1.0-35B
• MTP head source: mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit

Model Sources

• Base model: https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B
• Quantization tool: https://mlx-optiq.com
• Serving engine used for benchmarks: https://github.com/jundot/omlx

Model Properties

Uses

Direct Use

This model is intended for local text generation on Apple Silicon devices using MLX based inference engines such as mlx-lm or optiq serve. It is suited for users who want most of the quality of the BF16 checkpoint at roughly half the memory footprint. Note: the bundled MTP head does not currently provide working speculative decoding with this quantization (see Multi-Token Prediction).

Out-of-Scope Use

This is a quantized derivative of a third party base model. It has not been independently evaluated for safety, factuality, or fitness for any particular downstream task. It should not be used in high stakes settings (medical, legal, financial, or safety critical decisions) without additional evaluation. Refer to the base model card for its intended use cases and known limitations, since these are inherited by this quantization.

Bias, Risks, and Limitations

Quantization can shift a model's behavior relative to the original weights, even when overall benchmark scores are similar. Mixed precision quantization at 7.5 BPW is expected to be close to BF16 quality, but no independent evaluation of downstream task accuracy, factuality, or bias has been performed for this specific quantized artifact. Users should treat outputs as they would from the base model and are encouraged to run their own evaluations for their use case before deploying it in production.

The intelligence benchmark results reported below use 30-question samples from each dataset and should be treated as indicative rather than definitive. Full dataset evaluations may yield different figures.

How to Get Started with the Model

With mlx-optiq serve

optiq serve \
  --model programmer-666/Ornith-1.0-35B-oQ7 \
  --port 8080

With mlx-lm

mlx_lm.generate \
  --model programmer-666/Ornith-1.0-35B-oQ7 \
  --prompt "Your prompt here"

Note on MTP: This repository still contains a grafted MTP head (mtp.safetensors), but testing found it does not work correctly as a speculative decoding draft model with this oQ7 mixed-precision quantization. Do not pass a --mtp flag to optiq serve for this model. The file is retained for future compatibility work — see Multi-Token Prediction below for details.

Training Details

This repository does not modify the base model's weights beyond quantization; no additional fine-tuning was performed. For training data and training procedure, refer to the base model card.

Quantization Procedure

Quantization was performed with mlx-optiq, which runs a per-layer KL sensitivity analysis to assign bit widths. Layers with higher sensitivity to quantization error retain 8-bit precision, while more robust layers are assigned 6-bit precision. The reference model used during sensitivity calibration was the original BF16 checkpoint.

Conversion command:

optiq convert deepreinforce-ai/Ornith-1.0-35B \
  --candidate-bits 6,8 \
  --target-bpw 7.5 \
  --reference bf16 \
  --n-calibration 40 \
  --group-size 32 \
  --skip-baselines \
  -o Ornith-1.0-35B-oQ7-mtp

Multi-Token Prediction (MTP)

The mtp.safetensors file contains an auxiliary prediction head grafted from mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit. Architecturally, the graft is sound: Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share an identical structure (hidden dimension 2048, 40 layers, 256 experts, same block layout), and Ornith is fine-tuned directly from Qwen3.5-35B-A3B.

Known issue — MTP does not work with this oQ7 quantization. In principle, MTP uses this auxiliary head as a draft model for speculative decoding, giving roughly a 1.3x to 1.4x decode speedup on Apple Silicon during greedy generation. In testing, however, the head did not pair correctly with the oQ7 mixed-precision weights in this repository, so that speedup is not realized here and speculative decoding should not be enabled (e.g. do not pass --mtp to optiq serve).

Because of this finding, the repository was renamed from Ornith-1.0-35B-oQ7-mtp to Ornith-1.0-35B-oQ7 to avoid implying working speculative decoding support. The mtp.safetensors file itself has not been deleted and remains in the repo for future compatibility work.

Hardware Requirements

Evaluation

Testing Setup

All benchmarks were run locally on an M4 Max MacBook Pro (128GB unified memory) using oMLX, an LLM inference engine optimized for Apple Silicon. Three models were evaluated side by side:

• Ornith-1.0-35B-oQ7 (this repository)
• Ornith-1.0-35B-bf16 (original BF16 checkpoint, reference)
• Qwen3.6-35B-A3B-OptiQ-4bit (MTP head source, shown for context) Performance figures come from single request runs unless noted as continuous batching. Intelligence benchmarks were run with thinking mode enabled and use 30-question samples drawn randomly from each dataset; they are indicative and should not be compared directly to full-dataset evaluations published elsewhere.

Intelligence Benchmarks

All results are for this model (Ornith-1.0-35B-oQ7) with thinking enabled.

These results represent 30-question random samples from each dataset evaluated with thinking mode enabled. Sample-based scores carry higher variance than full dataset evaluations and may not reflect performance on the full benchmark. Results are not directly comparable to published leaderboard figures which typically use the full dataset without thinking mode.

Performance Benchmarks

Ornith-1.0-35B-oQ7 (this repository)

Continuous batching, pp1024/tg128:

Ornith-1.0-35B-bf16 (reference)

Continuous batching, pp1024/tg128:

Qwen3.6-35B-A3B-OptiQ-4bit (MTP head source, shown for reference)

Continuous batching, pp1024/tg128:

Summary

Compared to the BF16 checkpoint, this quantization roughly halves peak memory usage (37 to 54 GB versus 66 to 82 GB depending on context length) while producing 51% faster token generation at standard context lengths (94.2 tok/s versus 62.3 tok/s at pp1024/tg128). The BF16 model's continuous batching throughput degrades at batch size 2 (0.69x), while this quantization scales consistently to 1.33x at batch size 4. The 4-bit Qwen3.6-35B-A3B-OptiQ-4bit model is faster and lighter on its own; it is included here as the MTP draft head source and as a context point for the tradeoff between model size and throughput.

Licenses

• Ornith-1.0-35B (base model): MIT License
• Qwen3.5-35B-A3B (base architecture): Apache 2.0
• Qwen3.6-35B-A3B (MTP head source): Apache 2.0
• This repository (quantized weights and MTP head): Apache 2.0 Users should review the license terms of each underlying component before use or redistribution.

Citation

If you use this model, please cite the original Ornith-1.0-35B model and the mlx-optiq quantization tool.

@misc{ornith-1.0-35b-oq7,
  title = {Ornith-1.0-35B-oQ7},
  author = {programmer-666},
  year = {2026},
  note = {MLX mixed precision quantization of deepreinforce-ai/Ornith-1.0-35B; includes a grafted MTP head retained for future compatibility work, not currently functional with this quantization},
  howpublished = {\url{https://huggingface.co/programmer-666/Ornith-1.0-35B-oQ7}}
}

Model Card Contact

For questions about this quantization, open a discussion on this repository's Community tab. For questions about the base model, refer to deepreinforce-ai/Ornith-1.0-35B.