OmniCoder-9B-AWQ-W4A16-ASYM

OmniCoder-9B — AWQ W4A16_ASYM (INT4)

INT4 weight-only quantization of Tesslate/OmniCoder-9B (Qwen3.5 VLM architecture, hybrid linear/full attention) produced with llm-compressor 0.12.0.

• Size: 18 GB (BF16) → 8.1 GB on disk
• Scheme: AWQ, W4A16 asymmetric, group size 128, compressed-tensors / pack-quantized
• Calibration: 256 samples × 2048 tokens from ise-uiuc/Magicoder-Evol-Instruct-110K (code-domain calibration), via the model's chat template
• Kept in BF16 (ignore): vision tower (model.visual.*), lm_head, MTP head (mtp.fc), linear-attention gating projections (linear_attn.in_proj_a/b) — following the layer recipe validated in cyankiwi/OmniCoder-9B-AWQ-4bit. The untied 248k-vocab embeddings and lm_head dominate the remaining BF16 weight.

The exact reproducible recipe is in recipe.yaml.

Usage (vLLM)

vllm serve Ar4ikov/OmniCoder-9B-AWQ-W4A16-ASYM

vLLM executes the packed INT4 weights natively: expect roughly 9–10 GB of weight VRAM plus KV cache — fits a single 24 GB GPU (RTX 3090/4090) with room for context.

Note: loading with plain transformers decompresses weights back to BF16 in memory; the memory savings materialize in vLLM (or other compressed-tensors-aware runtimes).

Quantization code

Produced with AWQModifier(targets=["Linear"], scheme="W4A16_ASYM", ignore=[...]) and model.save_pretrained(..., save_compressed=True); see recipe.yaml for the full config.