Ornith-1.0-9B-OptiQ-6bit

mlx-community/Ornith-1.0-9B-OptiQ-6bit

A 6-bit mixed-precision MLX quant of ornith-ai/Ornith-1.0-9B, built on the Qwen3.5-9B architecture. Sensitive layers are kept at 8-bit and robust ones at 6-bit.

18.8 GB of bf16 weights become 9.06 GB, of which 8.15 GB is the quantized language tower and 0.91 GB the bf16 vision tower.

Built at the request of a discussion on the 4-bit card asking for a quality-first variant. If you are choosing between the two, this one costs 1.0 GB more on disk and puts every layer at least two bits above the 4-bit variant's floor.

Sizes here are decimal GB, matching what the Hub sidebar reports.

Image input works. The vision tower is kept at bf16 in a sidecar, so this quant takes images as well as text.

Quantization details

Following the naming convention llama.cpp uses for Q4_K_M and similar mixed quants, the "6-bit" label is the predominant precision, not the weighted average.

The three bpw figures differ for two separate reasons, both worth being explicit about. 6.626 is the weighted mean over the quantized tensors themselves. 7.284 adds what the packed format costs on disk: per-group scales and biases, plus 177 bf16 tensors (norms and biases, 1.06 M params) that are never quantized. 8.099 adds the bf16 vision tower, which is not quantized at all, by design, as in every OptiQ VLM.

Verified: 250 quantized tensors, U32 elements * (32 / bits) summing to exactly 8,952,741,888 params.

How the bit-widths were chosen

By structural rules (--method static), not by measurement. mlx-optiq assigns bits from architectural position: embeddings and output head, first and last block, attention over dense MLP. Same method as Qwen3.5-122B-A10B-OptiQ-2bit and Mistral-Small-4-119B-2603-OptiQ-2bit.

This differs from the 4-bit variant, which transferred a KL-divergence sensitivity map from Qwen3.5-9B. Neither is a sensitivity sweep measured on Ornith's own weights. We started one and abandoned it, for a reason worth stating:

At candidate bits 6 and 8, the measured per-layer KL gap between the two is 3-5%, against roughly an order of magnitude for 4 vs 8. Worse, the sweep on 32 GB of RAM has to run in uniform_4bit reference mode (mlx-optiq requires the bf16 checkpoint to fit in 70% of available RAM, so 18.8 GB of weights needs ~26.9 GB free), so KL is measured against a 4-bit baseline that is worse than both candidates. That works when one candidate is the baseline and you are measuring the gain from upgrading; at 6 vs 8 the difference sinks into calibration noise. An 8-hour sweep would have produced a layer ranking driven by noise, and a card claiming it was measured. Structural rules with an honest label are the better artifact.

If you want the measurement, it needs a machine that fits the bf16 reference.

Fidelity

Weight-level, measured by dequantizing every quantized tensor out of this artifact and comparing against the bf16 source tensor by tensor. No prompts, no sampling, hardware-independent.

Worst single-element error across all 250 tensors: 0.0117.

The 8-bit figure is the sanity check. Affine 8-bit at group size 64 lands at 42.6-42.7 dB across every architecture we have measured, and 42.59 dB here says the conversion is sound. The 0.742% relative error also matches the 0.7% the 4-bit card reports for its own 8-bit layers.

Throughput

M-series, 32 GB unified memory, mlx-lm text path:

The image path's 20.93 GB peak is on a 760x240 screenshot. Larger images cost more. Text-only fits comfortably in 16 GB; image work on 16 GB will be tight.

Verification

Run on the finished artifact, after conversion:

• Arithmetic. 17*24 gives 408, with correct intermediate steps.
• Instruction following. "List exactly three primary colors, one per line, no other text" gives three lines, no extra text.
• Code. Correct Python string reversal.
• Multilingual. Spanish prompt answered in Spanish, correctly.
• Image. A synthetic UI screenshot with two buttons and a total: names "SubmitOrder", "Cancel" and "Total: 42.50 USD", all three correct.
• Vision sidecar. Checked tensor by tensor against the sidecar published in the 4-bit repo: 333/333 tensors, identical names, identical shapes, all BF16.

What was not measured

• No task benchmarks. No MMLU, GSM8K, or any harness score. Nothing here claims this quant scores within N% of the source on any benchmark.
• No perplexity or top-1 agreement against bf16. The bf16 control does not fit alongside this model in 32 GB, and our aggregate-perplexity harness has a known defect where one out-of-domain passage dominates the total. Weight-level fidelity above is exact and prompt-independent; it answers a different question than behavioural quality, and does not substitute for it.
• No sensitivity measurement on Ornith's own weights. See above.
• No comparison against the 4-bit variant on the failure the requester reported (skipped instructions, shallower reasoning). Higher bit-widths and a 2.3x lower reconstruction error make it plausible this helps, but we did not measure that specific behaviour, and the fidelity table is not evidence for it.
• Long context untested. All checks were short prompts.

Usage

Text

The sidecar lives in an optiq/ subfolder, so a stock *.safetensors glob ignores it and mlx-lm sees a clean language model.

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Ornith-1.0-9B-OptiQ-6bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain the difference between TCP and UDP."}],
    add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

This is a reasoning model: it thinks inside <think>...</think> before answering, so give it enough max_tokens to finish.

Images

Image input needs mlx-optiq, which loads the bf16 vision sidecar and feeds the merged embeddings to the quantized language tower. Use 0.4.17 or newer. Versions up to 0.2.18 look for the sidecar only in the repo root, so they will not find it in optiq/, and their vision-tower prefix list has no visual. entry, which this architecture needs. Note that a bare pip install mlx-optiq may resolve to an older version depending on your other pins, so check optiq.__version__.

pip install "mlx-optiq>=0.4.17"

from PIL import Image
from optiq.runtime.engine import OptiqEngine

engine = OptiqEngine("mlx-community/Ornith-1.0-9B-OptiQ-6bit")
answer = engine.generate("What is in this image?",
                         images=[Image.open("photo.jpg")], max_tokens=512)
print(answer.text)

The base model ships no MTP head, so this quant has no speculative-decoding sidecar. OptiqEngine prints MTP weights not found on load and falls back to autoregressive decoding; that line is expected, not an error.

Sibling variants

Weight sizes, read from the Hub API (.safetensors only, decimal GB):

The two OptiQ rows include an identical 0.91 GB bf16 vision tower, so on the language tower alone this quant is 8.15 GB against the 4-bit's 7.10 GB. That 1.05 GB is the whole cost of moving the 4-bit floor up to 6.

The uniform variants do not carry a vision sidecar. If you need image input, use one of the two OptiQ quants.

Quantization does not change the behaviour or alignment of the base model. Use it under the same terms as the original.