Downloads · 30 days
35
37% of all-time downloads
JasonW2025/Qwopus3.6-35B-A3B-v1-PrismaQuant-5.06bit-MXFP6mix-vllm
Qwopus3.6-35B-A3B-v1-PrismaQuant-5.06bit-MXFP6mix-vllm is a machine learning model from JasonW2025. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Mixed-precision quantization of Jackrong/Qwopus3.6-35B-A3B-v1 (Qwen3.6 MoE, 35B total / ~3B active, hybrid linear+full attention, in-checkpoint MTP head) produced with PrismaQuant (spark-improvements fork, streaming p…
Downloads · 30 days
35
37% of all-time downloads
All-time downloads
94
Public
Parameters
23.6B
26.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors26.9 GB · 100%
How the weights are stored.
U818.8B · 80%
From the Hugging Face model README
Mixed-precision quantization of Jackrong/Qwopus3.6-35B-A3B-v1 (Qwen3.6 MoE, 35B total / ~3B active, hybrid linear+full attention, in-checkpoint MTP head) produced with PrismaQuant (spark-improvements fork, streaming production-render pipeline).
This is a experimental build with MXFP6 and exists only for testing and demostration purpose.
Same 5.06-bit budget as the sibling PrismaQuant-5.06bit-vllm, but with MXFP6_E2M3 added to the allocator menu (NVFP4 / MXFP6_E2M3 / MXFP8_E4M3 / BF16). On measured per-Linear error the allocator displaced MXFP8 entirely — fp6 dominates fp8 as the escalation tier at this budget.
Stock vLLM (as of 0.22.1) has no CUDA compressed-tensors scheme for
mxfp6-*-pack-quantized and fails with
No compressed-tensors compatible scheme was found. Serve with the
vllm_patches/mxfp6/ overlay from the PrismaQuant fork (a Python-only,
anchor-verified patch + derived Docker image):
# Tier-2 (recommended): fp6 expert groups stay packed on GPU (~6.5 bpp
# residency for the fp6 share) and dequantize inline in Triton fused-MoE
# kernels. Bit-identical math to Tier-1; faster and smaller.
PRISMAQUANT_MXFP6_TIER2=1 vllm serve \
JasonW2025/Qwopus3.6-35B-A3B-v1-PrismaQuant-5.06bit-MXFP6mix-vllm \
--quantization compressed-tensors --kv-cache-dtype fp8 \
--trust-remote-code --enable-prefix-caching \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
# Without the env knob the overlay serves Tier-1: fp6 groups are
# dequantized to BF16 once at load (BF16-class memory for that share).
Measured on a DGX Spark (GB10 / Blackwell sm_121, 121 GB unified memory), aggregate decode tok/s at concurrency 1/4/8:
| serving mode | resident weights | TPS 1/4/8 | MTP acc |
|---|---|---|---|
| Tier-1 (load-time dequant) | ~31 GiB | 57.9 / 128.9 / 187.8 | 67.8% |
| Tier-2 (packed + Triton inline dequant) | 25.2 GiB | 58.7 / 135.8 / 211.1 | 68.3% |
The NVFP4 share runs on vLLM's native kernels in both modes; only the 22 fp6 expert groups change path.