Qwen3-Next-80B-A3B-Instruct_Compressed-Tensors

Qwen3-Next-80B-A3B-Instruct — Quantized (compressed-tensors for vLLM)

This repository provides quantized runtime packages of
Qwen/Qwen3-Next-80B-A3B-Instruct, repackaged for vLLM using the compressed-tensors format.

TL;DR

• This repo is quantized with branches W4A16 and W8A16.
• Load with vLLM using --quantization compressed-tensors.
• Qwen3‑Next A3B is an 80B‑parameter hybrid MoE model that activates ~3B params per token and supports ultra‑long context (≈262K). Only a subset of experts is active at a time, but full weights still must be resident in GPU/CPU memory for fast inference.

---

What’s special about Qwen3‑Next (A3B Instruct)

• Hybrid MoE / A3B: 80B total params with ~3B activated at inference; experts are sparsely selected per token.
• Experts: 100s of experts with a small top‑k activated per token; includes a shared expert for stability.
• Context length: native ≈262,144 tokens (and beyond with certain frameworks).
• Instruction‑tuned variant (this repo) – optimized for stable, formatted chat responses (no “thinking” traces).

See the parent model card and official posts for detailed specs and benchmarks.

---

Revisions & Branches

The main branch is a landing page (model card + links). All runnable artifacts live under per‑revision branches.

• main — placeholder / landing page
• W4A16 — 4‑bit weights / 16‑bit activations builds and runtime assets
• W8A16 — 8‑bit weights / 16‑bit activations builds

Quick links:

• 🔗 main
• 🔗 W4A16
• 🔗 W8A16

---

Repository Contents (per revision)

• Sharded quantized weights in .safetensors with an index (model.safetensors.index.json)
• config.json including compressed‑tensors metadata (weight_format, quantization, quantization_config)
• Tokenizer artifacts (tokenizer.json, tokenizer.model, etc.)
• Optional: chat_template.jinja (inherits the parent finetune’s chat format)

Exact files can differ by branch; see the Files and versions tab for each revision.

---

Quantization recipe & Qwen3‑Next nuances (what this export does)

These builds were created with an AWQ W4A16 / W8A16 recipe using llmcompressor and a small WikiText calibration set. Important choices tailored to Qwen3‑Next A3B:

• Calibration data: wikitext-2-raw-v1 validation split; 64 samples, tokenized with the chat template; sequence length 1024.
• Format & group size: weight‑only INT4 with group_size=128 (A16 activations are runtime dtype); non‑power‑of‑two channels handled.
• FFN policy: do NOT ignore FFN projections (gate_proj, up_proj, down_proj) — they are quantized.
• MoE routing kept full‑precision: router/dispatcher linears left unquantized (e.g., names including router, expert_choice, dispatch, scores, route, topk, switch) for stable expert selection.
• Head left unquantized: lm_head remains in higher precision.
• MoE‑aware calibration: calibrate_moe_context=True to properly calibrate sparse‑expert activations.
• Symmetry: the W4A16 build here uses symmetric INT4 weights; activations are BF16/FP16 at inference (A16).
• Save: exported with save_compressed=True to write compressed‑tensors metadata for vLLM.

These design choices aim to preserve router stability and FFN fidelity in the A3B hybrid‑MoE layout while offering strong memory savings.

---

Quickstart — vLLM (compressed‑tensors)

Install vLLM (recent version recommended):

pip install vllm

Serve (adjust to your hardware):

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 vllm serve TheHouseOfTheDude/Qwen3-Next-80B-A3B-Instruct_Compressed-Tensors   --quantization compressed-tensors   --tensor-parallel-size 8   --max-model-len 262144   --gpu-memory-utilization 0.70   --dtype bfloat16

Query via Chat Completions:

curl http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "TheHouseOfTheDude/Qwen3-Next-80B-A3B-Instruct_Compressed-Tensors",
    "messages": [
      {"role":"system","content":"You are Qwen3-Next (A3B), helpful, precise, and safe."},
      {"role":"user","content":"Outline a retrieval pipeline for scientific PDFs."}
    ],
    "max_tokens": 512,
    "temperature": 0.7,
    "top_p": 0.95
  }'

Note: compressed‑tensors is a vLLM runtime format. Loading this artifact directly in vanilla 🤗 Transformers is not supported; use vLLM for inference. For Transformers, use a different export (e.g., GPTQ/AWQ compatible) or full‑precision weights.

---

Prompting / Chat Template

This package follows the parent finetune’s chat conventions. If a chat_template.jinja is present, apply_chat_template will use it automatically.

---

Lineage

• Finetuned parent: Qwen/Qwen3-Next-80B-A3B-Instruct
• This repo: Quantized child of the finetune (compressed‑tensors for vLLM)

---

Hardware & Tips (rule‑of‑thumb)

• 80B‑class MoE with A3B still requires housing all 80B weights in GPU/CPU memory, though only ~3B are active per token.
• Long contexts are KV‑cache heavy—tune --max-model-len and batch size.
• Prefer BF16 on GPUs with native support; otherwise FP16.
• Consider CUDA Graphs if stable in your stack.

---

License & Usage

This distribution inherits the licenses/policies of the finetuned parent model (Apache‑2.0).
Use of the model constitutes acceptance of the upstream terms.

---

Changelog

• v1 (current) — Quantized compressed‑tensors exports for Qwen3‑Next‑80B‑A3B‑Instruct; added W4A16‑ASYM and W8A16 branches; model card set for Quantized classification.