Ornith-1.0-9B-AWQ-W4A16-ASYM

Ornith-1.0-9B-AWQ-W4A16-ASYM

INT4 AWQ (W4A16 asymmetric, group size 128) quantization of deepreinforce-ai/Ornith-1.0-9B — a 9B dense agentic-coding model (Qwen3.5 / Gemma 4 lineage) with a hybrid linear-attention + full-attention backbone.

Produced with llm-compressor. Weights are physically packed to 4 bits (compressed-tensors) and load natively in vLLM on Ampere+ GPUs (Marlin kernels). This is a weight-only quant — activations stay BF16.

• Size: ~18 GB (BF16) → 8.1 GB on disk (single model.safetensors shard)
• Scheme: W4A16_ASYM, group size 128
• Calibration: 256 samples from ise-uiuc/Magicoder-Evol-Instruct-110K (code-domain), max sequence length 2048, applied through the model chat template
• Tooling: llm-compressor 0.12, transformers 5.10, calibrated on 2×RTX 3090

What is quantized

This is a hybrid model (mixed linear_attention / full_attention layers). The recipe follows the proven approach for this architecture: quantize the standard projections, keep the sensitive state-space gates and non-LLM parts in BF16.

Quantized to INT4 (200 Linear modules):

• full-attention q_proj / k_proj / v_proj / o_proj (8 layers)
• all MLP gate_proj / up_proj / down_proj (32 layers)
• linear-attention out_proj, in_proj_qkv, in_proj_z (24 layers)

Kept in BF16 (ignore, 159 Linear modules):

• the entire vision tower — re:model\.visual\..* (110)
• linear-attention SSM gates in_proj_a / in_proj_b (48) — sensitive to the recurrence
• lm_head, and any mtp head

Usage (vLLM)

vllm serve Ar4ikov/Ornith-1.0-9B-AWQ-W4A16-ASYM --trust-remote-code

Expected GPU memory ≈ checkpoint size + KV cache + activations.

Recommended sampling

The model emits <think>...</think> reasoning blocks before the final answer.

temperature = 0.6, top_p = 0.95, top_k = 20

Reproducibility

The recipe.yaml written by llm-compressor is included in this repo. config.json carries the quantization_config (compressed-tensors, format pack-quantized) describing the exact scheme, group size and ignore list.

Notes

• Quantizing lm_head and the vision tower hurts quality for little gain, so they stay BF16. Together with the large embedding table and the BF16 SSM gates, this is why a 9B hybrid compresses to ~8 GB rather than the ~4.5 GB of a pure-transformer 7B.
• Loading this checkpoint in plain transformers transparently decompresses the weights back to BF16, so VRAM there looks like the original model. The real INT4 memory saving is realized by vLLM's native 4-bit kernels.