Downloads · 30 days
410
8% of all-time downloads
a-ivanovitch/Qwen3-30B-A3B-Instruct-2507-AWQ-W4A16
Qwen3-30B-A3B-Instruct-2507-AWQ-W4A16 is a text generation model from a-ivanovitch. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Quality-focused AWQ quantization of Qwen3-30B-A3B-Instruct-2507 with GQA-aware attention preservation — 61GB → ~18GB.
Downloads · 30 days
410
8% of all-time downloads
All-time downloads
5.3K
Public
Parameters
5.4B
18.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors18.2 GB · 100%
How the weights are stored.
I3229B · 95%
From the Hugging Face model README
Quality-focused AWQ quantization of Qwen3-30B-A3B-Instruct-2507 with GQA-aware attention preservation — 61GB → ~18GB.
A carefully optimized AWQ (W4A16) quantization of Qwen's updated Mixture-of-Experts instruct model (July 2025). Unlike standard AWQ quantizations of this model, this version preserves full-precision attention layers to avoid the quality degradation caused by unprotected quantization of GQA attention projections.
Most AWQ quantizations of GQA models (including those from official sources) silently produce 48 warnings per run:
skipping AWQ for model.layers.X.self_attn.v_proj ... incompatible balance layers
This happens because AWQ's smoothing algorithm cannot coordinate scaling between v_proj (4 KV heads, 512 dim) and o_proj (32 Q heads, 4096 dim) in Grouped Query Attention architectures. The result: o_proj gets quantized to W4 without any activation-aware protection — the worst-case scenario for quantization quality.
We identified that when AWQ cannot perform its core function (activation-weighted channel protection) on the attention layers, it's better to keep them at full precision rather than apply blind quantization:
| Component | Standard AWQ | This model |
|---|---|---|
| q/k/v/o_proj (attention) | W4 without smoothing ⚠️ | BF16 full precision ✅ |
| gate/up/down_proj (MLP experts) | W4 with AWQ smoothing ✅ | W4 with AWQ smoothing ✅ |
| MoE routing gates | BF16 ✅ | BF16 ✅ |
| lm_head | BF16 ✅ | BF16 ✅ |
The MLP expert layers (gate_proj, up_proj, down_proj) — which represent 96%+ of the model's parameters — receive full AWQ treatment with both smoothing mappings intact:
post_attention_layernorm → [gate_proj, up_proj]up_proj → [down_proj]The attention layers (~1.3 GB) stay in BF16, adding minimal overhead for significantly better output quality.
| 🧬 Architecture | Qwen3-MoE — standard transformer with Mixture-of-Experts |
| 📐 Parameters | 30B total, 3B active per token (128 experts, top-8 routing) |
| 🗜️ Quantization | AWQ W4A16 (4-bit integer weights, 16-bit activations) |
| 📦 Size | ~18 GB (from 61 GB BF16) — ~70% reduction |
| 🔧 Format | compressed-tensors — native vLLM support |
| 📏 Context | 262,144 tokens natively |
| 🧠 Attention | BF16 full precision (GQA-aware preservation) |
| ⚙️ MLP Experts | W4 with full AWQ smoothing (18,432 modules) |
This is the quantized version of the July 2025 update, which brings significant improvements over the original Qwen3-30B-A3B:
Note: This model supports only non-thinking mode and does not generate
<think></think>blocks.
vllm serve Sophia-AI/Qwen3-30B-A3B-Instruct-2507-AWQ-W4A16
docker run --gpus all \
-p 8000:8000 \
vllm/vllm-openai:latest \
--model Sophia-AI/Qwen3-30B-A3B-Instruct-2507-AWQ-W4A16
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
model="Sophia-AI/Qwen3-30B-A3B-Instruct-2507-AWQ-W4A16",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain mixture-of-experts architectures in simple terms."},
],
max_tokens=512,
)
print(response.choices[0].message.content)
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"Sophia-AI/Qwen3-30B-A3B-Instruct-2507-AWQ-W4A16",
torch_dtype="auto",
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
"Sophia-AI/Qwen3-30B-A3B-Instruct-2507-AWQ-W4A16"
)
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What are the key improvements in Qwen3's July 2025 update?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
AWQ (Activation-Aware Weight Quantization) using llmcompressor with GQA-aware custom mappings and selective layer preservation.
| 📊 Samples | 256 |
| 📏 Sequence length | 1024 tokens (p90-optimized) |
| 🌍 Calibration language | Italian |
| 🔀 MoE coverage | All 128 experts calibrated (moe_calibrate_all_experts=True) |
| ⚙️ Pipeline | Sequential (layer-by-layer) |
| 🖥️ Hardware | 1× NVIDIA B200 SXM (192 GB VRAM) |
The following layers are kept in their original BF16 precision:
| Pattern | Count | Reason |
|---|---|---|
lm_head | 1 | Output projection — critical for token prediction |
mlp.gate | 48 | MoE routing gates — high impact on expert selection |
self_attn.q_proj | 48 | Query projection — GQA incompatible with AWQ smoothing |
self_attn.k_proj | 48 | Key projection — GQA incompatible with AWQ smoothing |
self_attn.v_proj | 48 | Value projection — GQA incompatible with AWQ smoothing |
self_attn.o_proj | 48 | Output projection — no AWQ smoothing possible without v_proj coordination |
A total of 241 modules are preserved in original precision. All remaining 18,432 MLP expert modules are quantized with full AWQ protection.
Qwen3-30B-A3B uses Grouped Query Attention (32 Q heads, 4 KV heads). AWQ smoothing works by coordinating scaling between consecutive layers:
v_proj (512 dim) ──scale──→ o_proj (4096 dim)
With GQA, the dimension mismatch (8:1 ratio) breaks this coordination. AWQ cannot compute proper per-channel protection for o_proj. Rather than quantize it without protection — which other quantizations do — we keep the entire attention block in BF16 for consistent precision.
| Setup | VRAM | Notes |
|---|---|---|
| 1× RTX 3090 (24 GB) | ~18 GB | ✅ Fits — AWQ W4 runs on all GPUs |
| 1× RTX 4090 (24 GB) | ~18 GB | ✅ Fits with room for KV cache |
| 1× RTX 5090 (32 GB) | ~18 GB | ✅ Comfortable fit |
| 1× A100 (40/80 GB) | ~18 GB | ✅ Plenty of headroom |
| 1× H100 (80 GB) | ~18 GB | ✅ Ideal for long contexts |
AWQ W4A16 runs on all NVIDIA GPUs (no compute capability requirement unlike FP8/NVFP4). This includes RTX 3090, 3080, and older cards.
Qwen3-30B-A3B is a standard transformer with Mixture-of-Experts (MoE) feed-forward layers:
The MoE architecture means 96%+ of parameters are in the expert MLP layers — exactly where AWQ smoothing is applied in this quantization.
compressed-tensors format is natively supported by vLLM. Other inference engines may require conversion.<think></think> blocks. For reasoning mode, use the Thinking variant.This model inherits the Apache 2.0 license from the base model.