gemma-2-9b-awq

gemma-2-9b-awq

AWQ (W4A16_ASYM) 4-bit quantization of Google's Gemma-2-9B, calibrated with llmcompressor, served with vLLM.

5.7 GB on disk · ~6 GB VRAM at load · 75 RPS on a single RTX 4090 · 200+ concurrent users · $0.03 per million tokens

I built this because I wanted to know: Can a 9B parameter model actually run in production on consumer hardware? The answer, after a lot of trial and error with CUDA OOMs, embedding dtype bugs, and locust async pitfalls, is a very solid yes — but only if you get the quantization right and understand where your bottleneck actually is.

---

Table of Contents

• Quick Start
• Performance
• Cost Economics
• Understanding the Bottleneck
• Quantization Details
• Bugs I hit so you don't have to
• How to Use
• The Full Serving Stack
• Why AWQ Instead of Alternatives
• Known Limitations
• Who built this

---

Quick Start

pip install vllm torch transformers

from vllm import LLM, SamplingParams

model = LLM(
    model="zaid646/gemma-2-9b-awq",
    quantization="compressed-tensors",
    max_model_len=8192,
    gpu_memory_utilization=0.92,
)

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = model.generate(
    ["Explain why AWQ quantization outperforms GPTQ for production serving."],
    params,
)
print(outputs[0].outputs[0].text)

Or with Docker:

docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai:v0.24.0 \
  --model zaid646/gemma-2-9b-awq \
  --quantization compressed-tensors

---

Performance

All numbers measured on a single NVIDIA RTX 4090 (24 GB) on Vast.ai, serving 200-token outputs. Zero failures across every test run.

AWQ vs FP16 Baseline

Scaling Behavior

Throughput caps at ~75 RPS (GPU compute bound for 200-token outputs at 27ms ITL). Beyond ~200 users, TTFT grows linearly with queue depth. The system never crashes — it degrades gracefully.

---

Cost Economics

Vast.ai RTX 4090: ~$0.45/hour

Bottom line: AWQ delivers 50x more tokens per dollar than FP16 on the same hardware, and 5x cheaper than OpenAI's cheapest model.

---

Understanding the Bottleneck

The most important finding of this project: We hit the compute ceiling (27ms ITL = GPU saturated) before the memory ceiling.

With AWQ using only 6 GB for weights, ~18 GB is free for KV cache. vLLM's PagedAttention manages this efficiently. The bottleneck is compute: each token takes 27ms to generate on the RTX 4090, and nothing short of better hardware (or FP8) can make that faster.

• Up to ~200 users: Throughput rises linearly with batch utilization
• 200-500 users: GPU saturated, queue builds, throughput plateaus at ~75 RPS
• 500-5,000 users: TTFT grows linearly with queue depth, ITL stays flat at 27ms

If your SLO requires TTFT < 500ms, this setup supports ~200 concurrent users. If TTFT < 2s is acceptable, ~500 users. Beyond that, you need more GPUs.

---

Quantization Details

• Algorithm: AWQ (W4A16_ASYM, group size 128)
• Calibration: 64 samples from ultrachat-200k via llmcompressor 0.12.1a20260701
• Output format: compressed-tensors (vLLM-native, no conversion)
• Base image: pytorch/pytorch:2.12.1-cuda12.6-cudnn9-runtime
• GPU cap during calibration: 11 GiB (CPU-offloaded ~7 GiB layers to avoid OOM)
• Calibration time: ~45 minutes on RTX 4090

The pipeline is sequential: process one transformer layer at a time, compute activation-aware scaling factors, quantize, offload. This keeps peak VRAM under 13 GB.

---

Bugs I hit so you don't have to

Bug 1: llmcompressor stable + Gemma-2 embedding dtype crash

RuntimeError: Expected tensor for argument #1 'indices' to have one of the
following scalar types: Long, Int; but got torch.cuda.FloatTensor

llmcompressor==0.12.0 passes float-typed input_ids to Gemma-2's embedding layer. Fix: Use a clean pytorch/pytorch base image (not vLLM's) with llmcompressor==0.12.1a20260701.

Bug 2: GraniteMoeParallelExperts import rename

The nightly renamed GraniteMoeParallelExperts to GraniteMoeExperts. Fix: One-line sed patch in Dockerfile.

Bug 3: CUDA OOM during AWQ calibration

device_map="auto" puts 18 GB model on 24 GB GPU, leaving ~6 GB for calibration buffers. Fix: max_memory={0: "11GiB", "cpu": "40GiB"} to reserve 13 GB for calibration.

Bug 4: async httpx silently fails in Locust

Locust's greenlet runtime doesn't provide an event loop for async httpx. Fix: Use synchronous requests with manual SSE stream parsing.

---

How to Use

With vLLM (recommended)

from vllm import LLM, SamplingParams

model = LLM(
    model="zaid646/gemma-2-9b-awq",
    quantization="compressed-tensors",
    max_model_len=8192,
    gpu_memory_utilization=0.92,
    max_num_seqs=128,
    enable_prefix_caching=True,
)

params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = model.generate(["Your prompt here."], params)
print(outputs[0].outputs[0].text)

OpenAI-compatible API

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zaid646/gemma-2-9b-awq",
    "prompt": "Explain AWQ quantization.",
    "max_tokens": 512,
    "temperature": 0.7
  }'

Docker Compose (full stack)

git clone https://github.com/ZAID646/llm-serving-lab.git
cp .env.example .env
# Edit .env with your HF_TOKEN
docker compose up -d

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "zaid646/gemma-2-9b-awq",
    device_map="auto",
    torch_dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained("zaid646/gemma-2-9b-awq")

Note: compressed-tensors is optimized for vLLM. Transformers inference works but throughput is significantly lower.

---

The Full Serving Stack

This model is one component of the llm-serving-lab project:

---

Why AWQ Instead of Alternatives

---

Known Limitations

1. TTFT degrades past ~200 users — GPU compute is the bottleneck, not memory
2. vLLM-optimized format — Transformers inference is slower
3. Measurable precision loss — <1% MMLU degradation but not lossless
4. No speculative decoding — Not yet stable with compressed-tensors
5. Single GPU ceiling — 75 RPS is the limit of one RTX 4090
6. Power draw — 4090 pulls ~350W under sustained load

---

Who built this

Mohammed Zaid Hussain — ML System Engineer.

I make LLMs work in production.

• GitHub
• Project repo
• Portfolio

If you found this useful, star the repo. If you found a bug, open an issue — PRs welcome.