Ornith-1.5-397B-EXL3-HQ-35bpw

Ornith-1.5-397B — EXL3 HQ (3.5 bpw)

A high-quality EXL3 quantization of ornith-ai/Ornith-1.5-397B, a 397B-parameter Mixture-of-Experts reasoning and agentic-coding model (MIT license) post-trained on top of Qwen 3.5 and Gemma 4.

This quant follows the same recipe as the earlier gigascake/Ornith-397B-EXL3-HQ-35bpw quant of Ornith-1.0 — same EXL3-HQ 3.5 bpw mixed-precision allocation, with all 512 experts preserved.

Quantized and benchmarked on 2 × NVIDIA RTX PRO 6000 (Blackwell Workstation, 96 GB each).

---

Quick facts

Bit allocation

Calibration: 250 rows × 2048 cols · Codebook: mul1 · Output scales: always

Model shape

60 layers · hidden size 4096 · 32 attention heads / 2 KV heads (GQA) · 512 experts, 10 active per token · MoE intermediate 1024 · vocab 248,320 · mixed linear/full attention (every 4th layer is full attention) · RoPE theta 10M.

---

Accuracy & quantization rationale

Generated directly from the full BF16 weights. This checkpoint was quantized from the unquantized ornith-ai/Ornith-1.5-397B BF16 source (740 GB, 122 files) — not from a previously-quantized or GGUF intermediate. The EXL3 --hq encoder calibrates against the BF16 tensors and assigns each layer the bitrate that minimizes its quantization error, so this is a single-generation lossy pass from the original precision rather than a re-quant.

The mixed-precision layout is the HQ encoder trading bits where they reduce error the most, against the 192 GB target:

• Experts at 3 bpw. The expert banks are ~99 % of the parameter count and are the most redundant part of an MoE — any single token only routes through 10 of 512 experts. They tolerate aggressive quantization with the smallest error-per-bit, so they absorb almost all of the compression. This is what makes a 397B MoE fit in this envelope at all.
• Attention, linear attention & shared expert at 5 bpw. These fire on every token and carry the routing signal, context mixing, and language quality. They are the most error-sensitive dense paths, so the HQ boost spends extra bits here to keep the per-token quantization error low — the highest-leverage place to spend a few extra bits.
• lm_head at 8 bpw. The head maps directly to the output logits over the full 248K vocab, so any error there distorts the entire token distribution. Keeping it near-full precision is the cheapest way to protect output quality.
• Vision tower unquantized (BF16), embeddings & norms unquantized. Stored at source precision, so zero quantization error on the vision path and the small normalization tensors.

Why this fits 192 GB so cleanly: pushing the cheap expert bulk down to 3 bpw and keeping only the error-sensitive dense paths (attention/router/head) at higher precision is what lets the full model, plus a 1,048,576-token FP8 KV pool for four concurrent 262K contexts, land on two 96 GB cards with ~1 GB + ~2 GB to spare. The bits follow the error.

What was measured here

No perplexity, KL divergence, or benchmark-vs-BF16 error was computed for this release. Validation covered structural integrity only, and it passed cleanly: no NaN/Inf tensors, all 60/60 layers present, all 512/512 experts present, tokenizer byte-for-byte identical to the source, and key config values preserved. See the "How well does it work?" section above for the (deliberately modest) quality framing.

---

Hardware & serving

Measured and served with ExLlamaV3 (tensor parallel across both cards). See ornith15-exl3-modelcard-perf.md for the full throughput table.

VRAM footprint: the full checkpoint plus a 1,048,576-token FP8 KV pool (four 262K contexts) fits on this GPU pair with headroom to spare — roughly ~95 GB + ~94 GB in use, leaving about 1 GB free on one card and 2 GB free on the other.

Example (TabbyAPI + ExLlamaV3)

--host 0.0.0.0 --port 5000 \
--model_dir /models/Ornith-1.5-397B-EXL3-HQ-35bpw \
--tensor_parallel_size 2 --cache_mode 8,8 \
--context_length 262144 --max_cache_size 1048576 \
--reasoning_parser qwen3 --tool_call_parser qwen3_xml

---

How well does it work?

Short answer: very well, subjectively — but this is not a benchmarked release.

This quant did well in a short, informal benchmark test I ran, but I have not run — and do not plan to run — a full benchmark suite (no SWE-Bench / Terminal-Bench / perplexity comparison against the unquantized weights is included here). The ~52 tok/s number above is a decode-throughput measurement only, not a quality score. Take the model on its own merits and evaluate it on your own tasks.

---

Notes

• Reasoning model: assistant turns open with a <think> block before the final answer. Serve with a --reasoning_parser qwen3 so the chain-of-thought is returned in a separate reasoning_content field.
• Tool calling: the model emits <tool_call> blocks that parse into the standard OpenAI-style tool_calls field — use --tool_call_parser qwen3_xml.
• Vision: the vision tower is stored unquantized, so the checkpoint can still accept image/video tokens; for text-only use it simply sits idle.
• Long context: YaRN is validated up to ~1M tokens (scaling factor 4.0) and is built into vLLM/SGLang.

---

Attribution & license

Quantization of ornith-ai/Ornith-1.5-397B, released by DeepReinforce AI under the MIT license. This EXL3 quant carries the same license forward. The mixed-precision HQ recipe matches the earlier Ornith-1.0 quant at gigascake/Ornith-397B-EXL3-HQ-35bpw.