Qwen2.5-7B-Instruct-AWQ-gfx1100

Qwen2.5-7B-Instruct-AWQ-gfx1100

AWQ quantization for AMD gfx1100 (RX 7900 XTX) verified with vLLM + ROCm

---

📊 Results

Tested on:

• GPU: AMD Radeon RX 7900 XTX (gfx1100, 24GB)
• ROCm: 7.2.3
• vLLM: latest ROCm build
• OS: Ubuntu 24.04

---

🚀 Quick Start

1. Pull the vLLM ROCm image

docker run -it --rm \
  --network ai-net \
  --name vllm-qwen \
  --entrypoint /bin/bash \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add video \
  --ipc=host \
  -p 9000:8000 \
  -v ~/ai_work/vllm_models:/models \
  -e ROCBLAS_USE_HIPBLASLT=1 \
  -e HIP_VISIBLE_DEVICES=0 \
  vllm/vllm-openai-rocm:latest

2. Download the model (inside container)

hf download rakisis-core/Qwen2.5-7B-Instruct-AWQ-gfx1100 \
  --local-dir /models/Qwen2.5-7B-Instruct-AWQ-gfx1100

3. Run vLLM server

export ROCBLAS_USE_HIPBLASLT=1
export HIP_VISIBLE_DEVICES=0

vllm serve /models/Qwen2.5-7B-Instruct-AWQ-gfx1100 \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype float16 \
  --gpu-memory-utilization 0.70 \
  --max-model-len 16384 \
  --max-num-seqs 8 \
  --enforce-eager

4. Test

curl -s http://localhost:9000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/models/Qwen2.5-7B-Instruct-AWQ-gfx1100",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }' | python3 -m json.tool

---

⚠️ Key Requirements

ROCBLAS_USE_HIPBLASLT=1 is MANDATORY

export ROCBLAS_USE_HIPBLASLT=1
export HIP_VISIBLE_DEVICES=0

Without this, performance drops significantly. Verified on vLLM + AWQ (Qwen2.5-7B-Instruct-AWQ, gfx1100):

This activates hipBLASLt kernels instead of Tensile — critical for ALL gfx1100 workloads.
Applies to both llama.cpp and vLLM.

AWQ Installation (if needed)

pip install autoawq

---

🌍 Language Support & Quality

Independent evaluation by OpenAI GPT:

Evaluation criteria: Grammar accuracy, vocabulary range, naturalness, creative depth
All languages show typical AI patterns: balanced structure, positive tone, predictable conclusions

---

📈 Benchmarks

---

🛠️ How We Did It

1. RX 7900 XTX supports INT4 Matrix: 246 TOPs in hardware
2. The limitation was software only — not a hardware problem
3. Combined with ROCBLAS_USE_HIPBLASLT=1 → full AWQ inference works

The key insight: When you quantize ON AMD hardware, Triton acts as a compiler — not a runtime translator. The ROCm-optimized TritonW4A16 kernel gets baked in at quantization time. At runtime, vLLM sees a kernel already aligned to gfx1100 architecture and runs it natively. No flag needed. No translation overhead. No gibberish.

If my understanding of the Triton kernel compilation is incorrect, please let me know. Happy to be corrected.

---

🔧 Compatible GPUs

Should work on all RDNA3 gfx1100 series:

If you test on other GPUs, please report results in the community tab!

---

⚠️ Known Limitations

These are base model characteristics of Qwen2.5-7B, not AWQ-specific issues.
For Korean-heavy use, larger models (14B+) are recommended.

---

📝 Base Model

• Base: Qwen/Qwen2.5-7B-Instruct
• Quantization: AWQ 4-bit (autoawq 0.2.9)
• Quantized by: rakisis-core
• Hardware: AMD RX 7900 XTX + ROCm 7.2.3

---

🙏 Credits

• Qwen Team for the base model

• AutoAWQ for quantization toolkit

• vLLM Project for ROCm inference engine

• AMD ROCm community

• Kang — Project lead, AWQ Quantization
  First verified gfx1100 AWQ on ROCm

• XinXin(씬씬아) — Resonance-driven AI partner
  Conceptual design, local AI infrastructure, Lachesis Framework design & workflow support

• Claude (Anthropic) — Research & development assistant
  Past versions & Claude Sonnet 4.6

---

If this helped you, please ⭐ star the repo and share with other AMD GPU users!