Ornith-1.5-35B-A3B-MagicQuant-GGUF

Ornith-1.5-35B-A3B — MagicQuant hybrid GGUFs

Per-group hybrid quantizations of ornith-ai/Ornith-1.5-35B-A3B (qwen3_5_moe, ~35B total / A3B active, 256 experts / 8 routed, vision-capable), produced by MagicQuant. The Q5 and Q6 tiers come from the August measured evolutionary search (v1); the Q4 tier was rebuilt on 2026-09-12 with the streamed-bytes allocator (v2) — see "Why the Q4 tier changed" below.

Pins: MagicQuant 3acccef (Q4 v2, Q6 v2), ed17991 (Q5), Foundry 2f99202.

Files

"Streamed per token" is the weight traffic one decode step touches, from the GGUF header: every non-expert tensor once, the routed experts at 8/256. On a bandwidth-bound machine (this was built and measured on an AMD Strix Halo, ~256 GB/s) it predicts decode speed better than file size does.

Why the Q4 tier changed (2026-09-12)

The August Q4 file spent its bits the wrong way round for decode: attention, shared expert and head were kept at BF16/F16 (trunk 11.86 bits/weight) while the routed experts — 97 % of the weights but only 8-of-256 touched per token — sat at 4.25. Each token therefore streamed 3.24 GiB, 84 % of it trunk. The rebuild holds the same payload (19.570 GiB, to the byte), moves the trunk to Q8_0 (8.66 bpw) and spends the freed bytes on the experts.

Everything below was measured in one session on one box, old and new file side by side, so the numbers are comparable to each other (not to the August table, which used a different run).

Allocation (bits per weight by tensor group, as chosen by the allocator): embeddings 8.5, attention Q/K/O/head 8.5, SSM/linear-attention 8.71, shared expert 6.6–7.6, routed experts 4.44 (MXFP4 floor, Q4_K/Q5_K upgrades on the most sensitive layers). Allocator settings: --budget-gb 19.57 --bandwidth-weight 0.00065 --probe-mode cumulative --floor X=MXFP4_MOE --floor E=Q8_0 --floor S,Q,K,O,H=Q8_0; storage utilisation 1.000; streamed bytes at λ=0 and at the chosen λ both 2.530 GiB (the floors, not the bandwidth term, set the outcome).

Read the KL row, not the PPL row, when comparing quality: at this chunk count PPL cannot separate the two files, KL can — and the rebuild is better on it, because the freed trunk bytes went to ffn_down_exps layers that the old uniform-MXFP4 allocation had left short.

Q5_K_M was left unchanged on purpose. It is uniform Q5_K with a 1.4 GiB trunk and already streams less per token than either Q4 file; there is no bandwidth win available at its size. If you want the fastest decode of the three and can afford 3 GiB more, take Q5_K_M.

The Q6 tier (2026-09-13)

Same method, at the Q6_K file's payload (28.80 GiB vs 28.83 GiB): trunk at Q8_0 instead of F16 head/embeddings and half-F16 attention, experts floored at Q6_K and upgraded with the freed bytes (6.83 bits/weight). Measured on the same instruments as the Q4 table, old and new side by side:

The decode numbers here come from two resident llama-server instances answering alternating requests, not llama-bench, so compare them within this table only (the Q4 table above used llama-bench). If you have the memory for it, Q6_K-v2 has the lowest KL to BF16 of the files measured this way (Q5_K_M was not KL-measured) and decodes faster than the old Q6_K; Q4_K_M-v2 remains the value pick.

How to read the older numbers honestly

The August search measured all rendered tiers on the same instrument (CPU llama.cpp perplexity, ctx 512, 100 chunks, wikitext-2; BF16 7.9691): Q4 7.9530, Q5 7.9951, Q6 7.9792 — a ±0.3 % spread that is noise at this resolution. Q3 (+2.96 %) and Q2 (+30.2 %) were searched and not rendered; that cliff is where quality is actually spent. The 2026-09-12 KL measurement is the sharper instrument that the August card said would be needed to separate tiers; it puts the new Q4 at 0.053 mean KL against BF16.

Notes

• Vision requires the mmproj file alongside the model.
• Chat template embedded and verified present in every file before upload.
• Vocab metadata (tokenizer.ggml.token_type) is INT32 per spec — these files load on current mainline llama.cpp builds (checked again on the rebuild).
• The Q4 rebuild uses MXFP4 blocks for most routed experts; any llama.cpp from mid-2026 handles them on CPU, CUDA, Vulkan and HIP.
• MoE with a shared-expert split; expect the usual llama.cpp requirements for qwen3_5_moe.
• Built and measured on an AMD Strix Halo (Ryzen AI MAX+ 395, gfx1151) host.