Ornith-1.5-35B-A3B-MQ4R

Ornith-1.5-35B-A3B — MQ4R quant for hipfire

Pre-quantized Ornith-1.5-35B-A3B (MoE, 35.9B total / 3B activated) in hipfire's .mq4r container:

embed_tokens at Q8F16; norms, A_log, dt_bias at F16; routed experts (gate_up_proj, down_proj) at uniform MQ4.

• Upstream model: ornith-ai/Ornith-1.5-35B-A3B (MIT)
• Engine: hipfire — a Rust-native LLM inference engine for AMD RDNA GPUs

Architecture (matches upstream config.json): 40 layers (10 full-attention + 30 DeltaNet linear-attention), 256 experts top-8, hidden_size 2048, head_dim 256, 262144 context, vocab 248320. Loaded by hipfire's arch_id=6 (qwen3_5_moe) path; the .mq4r wire extension triggers the automatic Redline PM4 route on single-GPU gfx1100/1151/1201 loads.

Note: the upstream checkpoint carries a vision tower and an MTP draft head; both are intentionally absent from this quant — hipfire's qwen3_5_moe text path does not consume them, which keeps the file at ~18.7 GB.

The recipe

MQ4R ("R" = reduced router): uniform 4.25-bit MQ4 across every weight class — attention, router, shared experts, conv1d, lm_head, routed experts. The --no-q8-router flag demotes the fixed tier (attention, lm_head, router, conv1d) from Q8F16 to MQ4, halving the dominant per-token decode byte count. embed_tokens stays at Q8F16 via its own fixed-tier arm.

Verification

Validated on RX 7900 XTX (gfx1100, RDNA3) with hipfire build b97a175c, served via a dedicated container with the Redline PM4 route and q8 KV.

Streaming decode throughput: 12 requests (4 long-form prompts × 3 rounds), 512 max tokens, thinking disabled (chat_template_kwargs.enable_thinking=false), exact token counts from server usage, first-token → last-token wall clock, warm-up run discarded. All 12 requests completed (GPU busy 100%, TTFT ≈ 30 ms):

Output verified coherent across prompts covering factual recall, creative writing, and technical explanation.

PM4 audit confirms full replay with zero fallback: allowlist_independent=40, no scratch-needing V2 kernels dispatched.

Known issue (uniform-MQ4 routed experts on this model family): when reasoning is enabled, if the token budget ends inside an open think span the daemon fails closed with open think span at end of generation. Disable thinking for throughput runs or give the budget enough headroom to close the span.

Usage

hipfire serve ~/.hipfire/models/ornith-1.5-35b-a3b-mq4r.mq4r --kv-mode q8 --idle-timeout 0

Requires an AMD RDNA GPU (gfx1100/1151/1201 class) with ROCm; the .mq4r extension selects the PM4 route automatically on single-GPU loads. With q8 KV the full 200k+ context fits a 24 GB card (~22 GB VRAM class). Pass --idle-timeout 0 for standing deployments so the model is never unloaded on idle.

Quantization format

All hipfire MagnumQuant (MQ) formats are FWHT-rotated (incoherence processing) with an 8-byte affine or fp16-codebook group header at group size 256. MQ4 = uniform 4.25-bits-per-weight scalar quantization. Reproduced with hipfire-quantize:

hipfire-quantize --format mq4v1 --no-kmap --no-q8-router --no-q8-conv1d --fixed-tier embed:q8 \
  --input <upstream-safetensors-dir> --output ornith-1.5-35b-a3b-mq4r.mq4r

<details> <summary>Why <code>--format mq4v1</code>?</summary>

The --format mq4v1 flag selects the V1 MQ4G256 kernel format (single-scale per group). This is required for full PM4 replay on GFX10/GFX11 — the V2 (MQ4G256V2) residual kernel needs scratch memory (private_segment_size=12, 2 VGPR spills) that PM4 dispatch cannot accommodate. When a single V2 kernel is encountered, hipfire calls gpu.replay.poison() and the entire PM4 replay path falls back to HIP (zero kernel overlap), costing ~30-40% decode throughput. The --no-kmap flag disables K-map precision promotion, keeping all non-expert weights at MQ4 instead of promoting some to MQ6 (which would also increase file size).

</details>

Attribution

• Base model: ornith-ai/Ornith-1.5-35B-A3B by ornith-ai — MIT license. All credit for the base model belongs to its authors; this repo redistributes only a lossy quantization of it.
• Quantization format & engine: hipfire — Rust-native, no-Python-hot-path inference engine for consumer + datacenter AMD GPUs.
• For reference SKUs of other models, see ghazni101/Apodex-1.1-mini-MQ4R and ghazni101/Nex-N2.5-mini-MQ4R.

Quantized locally on an AMD Ryzen 9 host from the upstream safetensors; validated on RDNA3 (gfx1100).