Downloads · 30 days
273
0% of all-time downloads
w341e/Qwen3.5-122B-A10B-abliterated-NVFP4
Qwen3.5-122B-A10B-abliterated-NVFP4 is a text generation model from w341e. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
NVFP4 (4-bit floating point) quantized derivative of wangzhang/Qwen3.5-122B-A10B-abliterated, which itself is derived from Qwen/Qwen3.5-122B-A10B.
Downloads · 30 days
273
0% of all-time downloads
All-time downloads
67.3K
Public
Parameters
71.2B
76.5 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors76.4 GB · 100%
How the weights are stored.
U858.7B · 82%
From the Hugging Face model README
NVFP4 (4-bit floating point) quantized derivative of wangzhang/Qwen3.5-122B-A10B-abliterated, which itself is derived from Qwen/Qwen3.5-122B-A10B.
This repository provides a modified derivative checkpoint for local inference and serving. The primary changes in this repository are NVFP4 quantization, weight repacking / export formatting, and serving compatibility adjustments.
| Property | Value |
|---|---|
| Intermediate Base Model | wangzhang/Qwen3.5-122B-A10B-abliterated |
| Original Base Model | Qwen/Qwen3.5-122B-A10B |
| Architecture | Qwen3.5 MoE (256 routed experts, 10B active) |
| Quantization | NVFP4 (compressed-tensors, nvfp4-pack-quantized) |
| Original Size | 228 GB (BF16) |
| Quantized Size | 71.2 GB (69% reduction) |
| Format | safetensors (2 shards) |
This model was quantized using a template-based weight replacement approach:
weight_packed and weight_scale) was regenerated from the abliterated BF16 weights using the reference checkpoint's weight_global_scale and input_global_scale values.config.json, quantization_config, global scales, and all metadata were preserved unchanged, ensuring full compatibility with vLLM's CUTLASS NVFP4 MoE kernel.| Component | Format | Notes |
|---|---|---|
| Routed experts (gate/up/down_proj) | NVFP4 | 256 experts × 48 layers × 3 projections |
| Shared experts | NVFP4 | 48 layers × 3 projections |
| Self-attention (q/k/v/o_proj) | NVFP4 | 12 full-attention layers |
| Linear attention | BF16 | 36 layers, kept at full precision |
| Embeddings, norms, gates | BF16 | Kept at full precision |
This model requires a text-only compatibility patch for vLLM since Qwen3.5 MoE is a multimodal architecture but this checkpoint contains only text weights.
# 1. Download the model
huggingface-cli download bjk110/Qwen3.5-122B-A10B-abliterated-NVFP4
# 2. Apply the text-only patch before starting vLLM
python vllm_patches/patch_qwen35_moe_text.py
# 3. Serve with vLLM
vllm serve /path/to/model \
--served-model-name Qwen3.5-122B-A10B-abliterated-NVFP4 \
--max-model-len 131072 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.90 \
--trust-remote-code \
--enable-prefix-caching \
--enable-chunked-prefill \
--reasoning-parser qwen3
### Docker Compose (Recommended)
A complete Docker Compose setup is provided in the `serving/` directory:
```bash
# Copy serving files
cp -r serving/ /path/to/your/vllm-setup/
# Edit .env to set MODEL_PATH
vim serving/.env
# Start
cd serving && docker compose --profile head up -d
See serving/ directory for:
docker-compose.yml — Full vLLM serving configuration.env.example — Environment variables templateentrypoint.sh — Entrypoint with automatic patch application| Configuration | Memory | max_model_len | Notes |
|---|---|---|---|
| 1× NVIDIA DGX Spark (GB10) | 121 GiB unified | 131,072 (128K) | Tested and verified |
| 1× GPU with 80+ GB VRAM | 80 GiB | ~65,536 | Estimated |
| Metric | Value |
|---|---|
| Throughput | 14.5 tok/s average, 16.8 tok/s peak |
| KV Cache | 222K tokens (20.4 GiB) |
| Max Concurrency | 6.16× at 128K context |
| Model Loading | ~13 min (2 shards) |