Ornith-1.0-35B-OptiQ-6bit

mlx-community/Ornith-1.0-35B-OptiQ-6bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs

A 6-bit mixed-precision MLX quant of deepreinforce-ai/Ornith-1.0-35B, a 35B sparse MoE built on the Qwen3.5-35B-A3B architecture. Sensitive layers are kept at 8-bit and robust ones at 4-bit.

70.2 GB of bf16 weights become 27 GB.

Image input works. The vision tower is kept at bf16 in a sidecar, so this quant takes images as well as text.

Quantization details

We follow the same naming convention llama.cpp uses for Q6_K and similar mixed-precision quants: the "6-bit" label is the predominant precision, not the weighted average.

The base model ships no MTP head, so this quant has no speculative-decoding sidecar.

Usage

Everything OptiQ-specific lives in an optiq/ subfolder, so a stock *.safetensors glob ignores it and mlx-lm sees a clean language model.

Serving

At 27 GB this is comfortable on a 36 GB Mac and fits smaller machines with SSD expert streaming, which keeps attention, the router and the embeddings resident and reads the routed experts from disk as the router picks them. optiq serve turns it on by itself when the model would not fit in RAM; --stream-experts forces it.

pip install mlx-optiq
optiq serve --model mlx-community/Ornith-1.0-35B-OptiQ-6bit --stream-experts

That gives you an OpenAI-compatible endpoint that accepts image content parts, with mixed-precision KV cache, tool-call healing and prompt caching.

Text

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Ornith-1.0-35B-OptiQ-6bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain the difference between TCP and UDP."}],
    add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

This is a reasoning model: it thinks before answering, so give it enough max_tokens to finish.

Images

Image input needs mlx-optiq, which loads the bf16 vision sidecar and feeds the merged embeddings to the quantized language tower. On a large MoE, serve it and send image content parts:

import base64, json, urllib.request

b64 = base64.b64encode(open("photo.jpg", "rb").read()).decode()
body = {"model": "x", "max_tokens": 512, "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64," + b64}},
    {"type": "text", "text": "What is in this image?"}]}]}
req = urllib.request.Request("http://127.0.0.1:8080/v1/chat/completions",
                             data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req))["choices"][0]["message"])

Verification

Text, arithmetic reasoning, and image understanding were all exercised on the finished artifact before release.

No task benchmarks were run on this quant; for measured quality numbers on the base architecture, see the Qwen3.5-35B-A3B OptiQ card.

Quantization does not change the behaviour or alignment of the base model. Use it under the same terms as the original.