Ornith-1.0-9B-AutoRound-GGUF

Ornith-1.0-9B AutoRound GGUF

GGUF quantizations of deepreinforce-ai/Ornith-1.0-9B, produced with Intel AutoRound using its alg_ext iterative calibration (~200 optimization steps per block) rather than plain round-to-nearest. At a given size this recovers noticeably more quality than vanilla GGUF quants, with the largest benefit at the low bit widths.

Ornith-1.0 is a self-improving, MIT-licensed family of open-source agentic-coding models from Deep Reinforce; the 9B is the lightweight, single-GPU member (dense, 32 layers, 256K context). See the original model card for capabilities and coding benchmarks. This repo only re-packages those weights as GGUF.

Quality vs. size

Every quant was measured against the BF16 reference on wikitext-2 test: PPL and, more importantly, KL-divergence of the output distribution (how far the quant drifts from full precision; lower is better). Top-1 is the share of positions whose argmax token still matches BF16. Reference BF16 PPL = 8.13.

Recommendations

• Q4_K_M ⭐ is the best all-round pick: smaller than the Q5 tier yet closest of the mid-size quants to BF16 (KLD 0.04, Top-1 91%). Start here.
• Q6_K / Q8_0 are effectively lossless (KLD under 0.004) when you have the VRAM.
• Q3_K_* are usable on tight memory. Reach for Q2_K_S only when you must, since 2-bit shows noticeable drift.

Why no Q5 or Q2_K_MIXED? On the KLD benchmark the Q5_K quants were not closer to BF16 than Q4_K_M despite being larger, so they add size without quality for this model and were omitted. The Q2_K_MIXED variant was defective here (broken output distribution) and was also excluded. (This base checkpoint ships no MTP/NextN head, so there is no speculative-decoding block to preserve.)

Multimodal

Ornith inherits a vision tower. Multimodal projectors are included for use via llama-mtmd-cli (pair any LM quant above with one of these). For text-only / coding use you can ignore them.

Usage (llama.cpp)

# chat / completion (the model's chat template, incl. <think> reasoning, is embedded)
llama-cli   -m Ornith-1.0-9B-Q4_K_M.gguf -ngl 999 -c 8192 -p "Write a Python LRU cache."

# OpenAI-compatible server
llama-server -m Ornith-1.0-9B-Q4_K_M.gguf -ngl 999 -c 8192

# multimodal (image + text)
llama-mtmd-cli -m Ornith-1.0-9B-Q4_K_M.gguf \
               --mmproj mmproj-Ornith-1.0-9B-f32.gguf \
               --image picture.jpg -p "Describe this screenshot."

How these were made

auto-round --model deepreinforce-ai/Ornith-1.0-9B \
           --scheme gguf:q4_k_m --enable_alg_ext --enable_torch_compile \
           --output_dir ./quantized

All outputs were NaN-scanned and inference-checked. The comparison table and the Pareto plot above (pareto.png, comparison.csv in this repo) come from llama-perplexity --kl-divergence against the BF16 reference.

Support

These quantized models are made in my spare time using expensive hardware such as DGX Spark systems for quantization and validation. If you find these GGUFs useful for your projects, consider buying me a coffee to help cover hardware and compute costs. Every bit of support helps me keep producing high-quality quantized models for the community!

☕ Support me on Ko-fi

Credits & license

• Base model © Deep Reinforce, MIT (LICENSE).
• Quantization: Intel AutoRound (Apache-2.0) and llama.cpp.