Downloads · 30 days
32
1% of all-time downloads
cyburn/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-PrismaQuant-8bit-vllm
Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-PrismaQuant-8bit-vllm is a text generation model from cyburn. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
[](https://github.com/RobTand/prismaquant) [](https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE) [](https://docs.vllm.ai/en/latest/features/quantization/compressedtensors.html)
Downloads · 30 days
32
1% of all-time downloads
All-time downloads
4K
Public
Parameters
27.1B
42.7 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors42.7 GB · 100%
How the weights are stored.
BF1615.5B · 57%
From the Hugging Face model README
Mixed-precision quantization of Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled produced by PrismaQuant — a per-Linear sensitivity-driven allocator that chooses each Linear module's format individually under a total-bit budget.
The source model is a reasoning distillation of Qwen3.6-35B-A3B trained on Claude 4.7 Opus chain-of-thought traces. It retains the full multimodal (vision + text) and MTP speculative-decoding head of the base architecture, with added reasoning capability. Enable thinking mode at inference time (see Serving below).
Why "every layer refracts into a different format": a naive uniform quantization either leaves disk on the table (keeping everything BF16 "to be safe") or loses quality (quantizing sensitive layers to 4-bit). PrismaQuant measures the actual Fisher-weighted MSE for every (Linear, format) pair and runs a multi-choice knapsack under a total-bit budget, so every bit lives where it buys the most likelihood.
| Metric | BF16 source | This artifact |
|---|---|---|
| Size on disk | ~70 GB | ~40 GB |
| Average bits per param | 16 | 8 |
| Reasoning (Claude 4.7 Opus distillation) | ✓ | ✓ |
| Multimodal (vision + text) | ✓ | ✓ |
| MTP speculative decoding heads | ✓ | ✓ |
Loads in vLLM (stock compressed-tensors) | ✓ | ✓ |
| Runtime backend | any | vLLM only |
This checkpoint uses three precisions, selected per-Linear by the allocator from measured sensitivity — not chosen uniformly. The visual encoder is kept entirely at BF16 (--visual-format=BF16), preserving full visual fidelity at the cost of a small disk overhead vs a quantized visual tower.
Allocator result (text body, 401 Linears):
| Format | Bits | Count (body alloc) |
|---|---|---|
| NVFP4 | 4.50 | 56 Linears |
| MXFP8_E4M3 | 8.25 | 29 Linears |
| BF16 | 16.00 | 193 Linears |
Export recipe (all modules, 511 entries including MTP + visual passthrough):
| Format | W | A | Use | Count |
|---|---|---|---|---|
| NVFP4 | 4-bit (FP4, group_size=16 with per-group FP8 scale + per-tensor global) | 4-bit (dynamic) | Lower-sensitivity MoE per-expert + medium-sensitivity dense Linears | 60 entries (9 dense + 48 per-expert + MTP) |
| MXFP8 | 8-bit (E4M3, group_size=32 with per-group E8M0 scale) | 8-bit (dynamic) | High-sensitivity dense Linears the allocator won't risk at 4-bit | 55 Linears |
| BF16 | 16-bit | 16-bit | Router, norms, biases, embed / lm_head, pos_embed, full visual encoder (110 Linears), highest-sensitivity MoE per-expert (32 entries) | 396 entries |
The allocator couples gate_up_proj / down_proj siblings so they share one per-tensor global scale (vLLM's FusedMoE requires this), and fused attention siblings (q_proj/k_proj/v_proj) share one per-tensor global scale so the packed qkv_proj loads without the "accuracy mismatch" warning. 112 fused-sibling groups were resolved during allocation.
61 Linears were ineligible for MXFP8_E4M3 due to kernel shape constraints and were restricted to the NVFP4 / BF16 choice.
On every NVFP4 weight the exporter runs, in order:
grid=32 candidate scales spanning [0.5·s₀, 1.5·s₀], round each weight to its nearest codebook neighbor at every candidate scale, pick the (scale, rounding-set) configuration minimizing activation-weighted per-group MSE sum_j a_j² · (w_orig,j - w_q,j)². Improve-or-keep gate against the post-GPTQ weight.Measured per-Linear output-MSE vs RTN baseline (geomean):
| Pipeline variant | out_mse ratio vs RTN |
|---|---|
| RTN (no passes) | 1.00 |
| GPTQ only | 0.41 |
| GPTQ + act_round polish (prior pipeline) | 0.99 (act_round undid GPTQ) |
| scale_sweep only | 0.33 |
| GPTQ + scale_sweep (this artifact) | 0.33 |
AWQ's γ-fold is not applied. On NVFP4's 16-channel groups, AWQ's per-channel rescaling pushes mixed-scale values into the same group and inflates per-group quant noise rather than reducing it.
The allocator sweeps a range of bit targets and reports predicted Δloss (Fisher-weighted MSE proxy). Target 8.0 bpp was chosen; the allocator suggested 4.5 bpp as the knee. The 8.0 target falls between the 7.0 and 8.25 table rows (achieved=0.695, NVFP4=56, MXFP8=29, BF16=193).
| Target bpp | NVFP4 | MXFP8_E4M3 | BF16 | Δloss (pred) |
|---|---|---|---|---|
| 4.50 | 246 | 1 | 31 | 1.154e+01 |
| 4.60 | 183 | 24 | 71 | 1.046e+01 |
| 4.70 | 140 | 40 | 98 | 1.014e+01 |
| 4.75 | 125 | 43 | 110 | 1.002e+01 |
| 4.85 | 96 | 36 | 146 | 9.843e+00 |
| 5.00 | 101 | 44 | 133 | 9.457e+00 |
| 5.25 | 98 | 47 | 133 | 8.677e+00 |
| 5.50 | 102 | 46 | 130 | 8.299e+00 |
| 6.00 | 87 | 38 | 153 | 7.453e+00 |
| 7.00 | 69 | 35 | 174 | 5.726e+00 |
| 8.0 | 56 | 29 | 193 | ← chosen |
| 8.25 | 54 | 33 | 191 | 3.572e+00 |
q_proj, k_proj, v_proj, o_proj): mixed NVFP4 / MXFP8 / BF16 per-Linear by sensitivityin_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj): same — note in_proj_a / in_proj_b variants were ineligible for MXFP8 due to shape constraints and resolved to NVFP4 or BF16gate_up_proj, down_proj): mixed per-expert precision — 48 per-expert NVFP4 + 32 per-expert BF16 = 80 total per-expert entries; the allocator assigned BF16 to the highest-sensitivity experts and NVFP4 to the rest. Joint per-tensor scale across each gate_up pair satisfies vLLM's FusedMoE requirement.mlp.gate): always BF16 (tiny, sensitive)Speculative-decoding head (1 layer) + its own MoE block quantized under the same per-Linear policy (3 089 MTP tensors merged at export). --speculative-config method=mtp drafts at the same precision as the body.
All 110 visual Linears are kept at BF16 via --visual-format=BF16. This is a deliberate conservative choice — the visual tower is left at full precision rather than running sensitivity-driven allocation on it. Disk overhead vs a quantized visual tower is small and visual quality is fully preserved.
lm_head — kept at BF16. vLLM's ParallelLMHead only accepts a single weight parameter; the compressed-tensors runtime rejects a compressed lm_head with KeyError: lm_head.input_global_scale. This is a vLLM runtime limitation, not a PrismaQuant design decision. The allocator would have chosen a lower precision here.embed_tokensmodel.visual.pos_embed (Parameter/Embedding, not an nn.Linear)This artifact is only runnable via vLLM's stock compressed-tensors support. vLLM 0.11+ is required.
vllm serve cyburn/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-PrismaQuant-8bit-vllm \
--trust-remote-code \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
This model was distilled from Claude 4.7 Opus chain-of-thought traces. Enable reasoning at inference time via the chat template:
messages = [{"role": "user", "content": "..."}]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
enable_thinking=True, # enables <think>...</think> CoT blocks
)
Or pass /think at the start of your prompt in interactive use. The model works without thinking mode enabled — it will respond directly — but reasoning quality is higher with it on.
VLLM_USE_FLASHINFER_NVFP4=1 to make the preference explicit.n=3 is the measured optimum for this family (n=2 leaves ~10 % tok/s on the table, n=4 regresses).image-text-to-text chat API — no special flags.Full pipeline is in the PrismaQuant repo:
--visual-format=BF16.ParallelLMHead does not register the NVFP4/MXFP8 compressed-tensors schemes. Costs a small amount of disk vs full quantization.rdtand/Qwen3.6-35B-A3B-PrismaQuant-4.75bit-vllm) could reduce the visual tower to ~22 % of its BF16 size.n=3 unless you verify against the draft-head acceptance-rate trace.@software{prismaquant2026,
title = {PrismaQuant: per-Linear sensitivity-driven mixed-precision
quantization for LLMs},
author = {Tand, Rob},
year = 2026,
url = {https://github.com/RobTand/prismaquant},
}