Ornith-1.5-35B-A3B-XYZ

Ornith-1.5-35B-A3B — XYZ Quantizations

Balanced GGUF quantizations of Ornith‑1.5‑35V (a 35B total / ~3B active hybrid SSM+attention Mixture‑of‑Experts model), built with the XYZ method: a balanced, MoE‑aware allocation in which attention layers and expert weights are kept at comparable precision rather than starving one for the other.

<div align="center"> <img src="https://huggingface.co/quimmedes/Ornith-1.5-35B-A3B-XYZ/resolve/main/Ilustration.png" width="420"/> </div>

Highlights

• Balanced allocation — routed experts (≈93% of parameters) and attention are quantized to similar effective bit‑rates; no "maxed attention / starved experts" skew.
• Int8 dense core — all non‑expert projections (attention q/k/v/output, SSM out/gate, router gate) kept at Q8_0; token embedding and output head also Q8_0.
• BF16 invariants preserved at every level: SSM α/β scalars, attention k/v, router input — negligible cost, maximal quality.
• Imatrix‑guided low‑bit expert quantization (IQ2 / IQ3 / Q4_K tiers) for accuracy at small sizes.

Files

Usage (llama.cpp)

# example: run the Q4 build fully on GPU
./llama-cli -m Ornith-1.5-35B-A3B-Q4-XYZ.gguf -ngl 99 -c 8192

Running on limited VRAM (8 GB GPU + 16 GB RAM)

Ornith is a MoE model (~3B active of 35B total), so it runs well by keeping the big routed experts in system RAM and putting only attention / SSM / shared parts on the GPU. Pin the experts to CPU with --n-cpu-moe, use a modest context, and quantize the KV cache to Q4_0 to cut VRAM.

# ~8 GB VRAM target: routed experts in 16 GB RAM, attention+SSM on GPU, q4_0 KV cache
./llama-server -m Ornith-1.5-35B-A3B-Q2.5-XYZ.gguf \
  -ngl 999 -ncmoe 30 \
  -c 8192 -ctk q4_0 -ctv q4_0 \
  --temp 0.6 --threads 4 --parallel 1

Notes:

• -ncmoe N keeps the routed MoE experts on CPU (use a small value like 30); only attention / SSM / shared-expert / embeddings go to the GPU. This is what makes an 8 GB card usable for a 35B model.
• -ctk q4_0 -ctv q4_0 quantizes the K/V cache to Q4_0, roughly halving its VRAM footprint so longer contexts fit in 8 GB.
• If it OOMs, lower -c (e.g. 4096) or increase -ncmoe value or use a smaller level; if you have headroom, raise -c.

Method / provenance

Built from the BF16 master (Ornith‑1.5‑35B‑BF16.gguf, 71 GB) using llama.cpp llama-quantize with per‑tensor‑type rules plus an importance matrix (blocks 0–39). The MTP/speculative block is quantized to Q4_K.