qwen3-coder-30b-a3b-reap50-nvfp4

Qwen3-Coder-30B-A3B · REAP-50% · NVFP4

A 50%-expert-pruned and NVFP4-quantized build of Qwen/Qwen3-Coder-30B-A3B-Instruct, made to run fully GPU-resident on a 16 GB Blackwell card (RTX 50-series / RTX PRO 6000, SM120) with real context to spare.

• Weights: 9.9 GB (single safetensors shard)
• Architecture: Qwen3-MoE, 64 experts/layer (down from 128), top-8 routing unchanged, 48 layers
• Format: NVFP4 4-bit weights (group size 16) + FP8 KV cache; MoE routers kept high-precision

How it was made

1. Prune — Cerebras REAP @ 50%. Each expert is scored by router-gate × activation-norm over a calibration set, and the 64 lowest-saliency experts per layer are removed, with router renormalization. The calibration set is personal: ~100 real coding-agent transcripts plus evol-codealpaca and SWE-smith trajectories — so the surviving experts are the ones this workload actually exercises.
2. Quantize — NVIDIA TensorRT Model-Optimizer 0.45, NVFP4. PTQ calibrated on the same personal transcripts (100 samples @ 8192 tokens). All 48 mlp.gate routers are excluded from quantization (kept in high precision — routers are too sensitive for 4-bit).

Evaluation

EvalPlus (greedy), measured on the pruned bf16 checkpoint.

Usage (vLLM, SM120)

Requires a Blackwell GPU (compute capability 12.0) and vLLM ≥ 0.24 (older builds lack SM120 NVFP4 kernels).

uv pip install -U vllm
VLLM_ATTENTION_BACKEND=TRITON_ATTN \
vllm serve AminFa12/qwen3-coder-30b-a3b-reap50-nvfp4 \
  --max-model-len 32768 --gpu-memory-utilization 0.9 --enforce-eager \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

First-launch warmup: FlashInfer JIT-compiles its NVFP4 kernels for SM120 on the first run — this can look like a hang (GPU idle, one CPU core busy) for several minutes before the server binds. Be patient the first time; it caches afterwards. --enforce-eager skips the separate torch.compile/CUDA-graph phase.

from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
print(c.chat.completions.create(
    model="AminFa12/qwen3-coder-30b-a3b-reap50-nvfp4",
    messages=[{"role":"user","content":"Write a Python function to merge two sorted lists."}],
    temperature=0.7, top_p=0.8).choices[0].message.content)

Credits & license

• Cerebras REAP — pruning method
• Qwen3-Coder — base model (Apache-2.0)
• NVIDIA TensorRT Model-Optimizer — NVFP4 PTQ

Licensed Apache-2.0, inherited from the Qwen3-Coder base model.