Downloads · 30 days
1.2K
37% of all-time downloads
malaiwah/Qwen-3.6-35B-A3B-SIQ-K6-K4K3
Qwen-3.6-35B-A3B-SIQ-K6-K4K3 is a machine learning model from malaiwah. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for trellis. The card lists the license as apache-2.0.
A lightweight, structurally faithful dev proxy for the GLM-5.2 SIQ serving stack — Qwen3.6-35B-A3B (VL MoE) with routed experts quantized as mixed Trellis K3/K4 in the exact per-layer partition geometry of willfalco/G…
Downloads · 30 days
1.2K
37% of all-time downloads
All-time downloads
3.2K
Public
Parameters
10B
21.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors19.9 GB · 100%
How the weights are stored.
I166.9B · 70%
From the Hugging Face model README
A lightweight, structurally faithful dev proxy for the GLM-5.2 SIQ serving
stack — Qwen3.6-35B-A3B (VL MoE) with routed experts quantized as
mixed Trellis K3/K4 in the exact per-layer partition geometry of
willfalco/GLM-5.2-EXL3-TR3-3.36bpw, served with online Trellis K6
conversion of eligible BF16 dense matrices on the vLLM gilded-gnosis +
SparkInfer runtime.
This will not make Qwen3.6 any smarter — that is not the goal. The goal is a test model light enough to load in minutes on one 32 GB GPU, yet representative enough (same 256-expert tier space, same mixed-K runtime contract, same loader/kernel paths) that changes to vLLM-GG, SparkInfer (b12x) and the SIQ method can be smoke-tested without touching a GLM-5.2 production deployment or waiting for ~330 GiB of weights to load.
"SIQ" (SparkInfer Quantization) is the honest name for the format the community ships as "EXL3-TR3": it uses the ExLlamaV3 Trellis/MCG math (quantized with exllamav3 0.0.43's LDLQ pipeline) but the rank-sliced checkpoint layout requires the custom vLLM + SparkInfer runtime — it will not load in ExLlama, Transformers, or stock vLLM.
| GLM-5.2-EXL3-TR3-3.36bpw (prod) | this model | |
|---|---|---|
| routed experts / layer | 256, top-8, 1 shared | 256, top-8, 1 shared |
| mixed partitions | layer 3: 206 K3 + 50 K4; layers 4–77: 160 K3 + 96 K4 | layer 0: 206 K3 + 50 K4; layers 1–39: 160 K3 + 96 K4 |
| expert bpw mean | 3.3726 | 3.3705 |
| K3/K4 as runtime kernel args (SparkInfer #117) | yes | yes (same one-grid kernels, second geometry) |
| dense/attention/router/shared/embeddings | BF16 (+ online K6 at load) | BF16 (+ online K6 at load) |
| KV cache | nvfp4_ds_mla (MLA) | fp8 (GQA/GDN — MLA dtype n/a) |
| MTP layer | quantized overlay (K3) | quantized (mixed K3/K4, layer-0 partition) — see V2 section |
| attention architecture | MLA + DSA | hybrid GDN (30) + GQA (10) — like Qwen3.6-27B |
| vision | — | tower carried BF16, vision requests work |
| weights on disk | ~326 GiB (TP4) | ~21 GiB (TP1) |
| loads on | 4× 96 GB | 1× RTX 5090 32 GB |
| Metric | value |
|---|---|
| Warm model load (online-K6 cache hit) | 10.1 s (17.88 GiB) |
| VRAM total @ util 0.92 | 30.3 GiB |
| GPU KV cache | 711,243 tokens @ 262,144 native ctx (fp8 KV) |
| Decode CC1 | 158.7 tok/s |
| Decode CC8 aggregate | 873 tok/s |
| Prefill (uncached) | 13,504 tok/s @ 31k prompt · 4,457 tok/s @ 249k prompt |
| Smoke tests | text w/ thinking ✓ · vision (image_url data URI) ✓ |
Reference point: the GLM-5.2-EXL3-TR3-3.36bpw production stack this mimics runs 91.8 tok/s CC1 / 314.9 CC8 / 2.4k–2.0k tok/s prefill on 4× RTX PRO 6000. The proxy loads in seconds and iterates faster than prod — which is its entire purpose.
| model | truncated-KLD vs BF16 (lower = closer) |
|---|---|
| this build (SIQ K6+K4K3) | 0.0105 |
| nvidia/Qwen3.6-35B-A3B-NVFP4 | 0.0186 |
Method: top-512 truncated KLD renormalized over the BF16 reference's
support; 2,044 positions of the Apache-2.0 LICENSE text in 512-token
chunks; prompt-logprob probes, offline vLLM r25, identical treatment for
all three engines (tools/kld_probe.py / tools/kld_compute.py). A single
fixed text — a data point, not a benchmark.
--moe-backend b12x,
validated on this stack) vs this SIQ build's 158.7 tok/s — while SIQ
measures 1.8× closer to BF16 (KLD 0.0105 vs 0.0186). Pick your axis.json_schema-constrained battery under speculation → 0 schema
violations, 0 parse failures (baseline arm also 0 violations). No
grammar bypass reproduced on r25 at k=2.MANIFEST.sha256 covers every
shipped file.The MTP layer's 256 routed experts are ALSO quantized (mixed K3/K4,
partition inherited from layer 0 as the runtime requires): the model-mtp
shard is 0.38 GiB instead of 1.57 GiB. Calibration is a disclosed
proxy: verified layer-39 activations routed through the MTP's own gate —
sound for a draft layer, whose quantization error can only lower
acceptance rate, never corrupt output (the target verifies every draft).
Speculative serving works (add to the serve command):
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
--compilation-config '{"cudagraph_mode":"NONE"}' --max-num-seqs 16
Measured: mean acceptance length 2.35-2.38 / 3 (k=2) on RTX 5090.
Two operator notes: (1) CUDA graphs MUST be off under speculation — the
GDN spec path hits q_len_per_req is part of the frozen cudagraph shape
(the same frozen-shape bug class documented on GLM-5.2 production, now
reproduced on the qwen3_5 path — which is this proxy doing its job);
(2) speculation is NOT a throughput win on a 3B-active model at TP1
eager (CC1 ~52 tok/s vs 158.7 without) — launch overhead dominates tiny
per-step math. It exists to exercise the production draft path, and it
does.
An attempted layer-0 quality upgrade (K3/K4 -> K4/K6, then K4/K5) mapped a hard kernel boundary instead: the one-grid mixed-Trellis kernel's shared -memory footprint at this geometry follows
smem_bytes = 93,184 + 8,192 x ((k_lo + k_hi) - 7) [layout trellis3_t256]
measured: (3,4) = 93,184 fits | (4,5) = 109,568 exceeds | (4,6) = 117,760 exceeds the RTX 5090's 101,376-byte opt-in limit. On 99 KB-smem consumer parts, (3,4) is the working ceiling; (3,5) computes to exactly 101,376 (boundary-exact, untested); K6 mixed tiers are B300-class (227 KB) territory. Encoding at K5/K6 works fine — serving is the constraint. Filed for the SparkInfer folks as tier-pair smem scaling data.
Also verified: the r25 loader does not consume non-empty keep_nvfp4
tier-bitmap entries (provenance metadata only) — a keep-NVFP4-within-SIQ
hybrid needs loader-side support first.
Requires the gilded-gnosis r25 image + the 2-file qwen3_5 rank-sliced loader
patch (in tools/patches/, submitted upstream):
podman run -d --name qwen36-siq --device nvidia.com/gpu=all --shm-size 16g \
-p 8000:8000 -v <this-repo>:/model:ro -v jit-cache:/cache \
-v <patched qwen3_5.py>:/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py:ro \
-v <patched qwen3_5_mtp.py>:/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5_mtp.py:ro \
-e VLLM_EXL3_ONLINE_TRELLIS_BITS=6 \
-e VLLM_EXL3_ENCODER_SOURCE=/opt/exllamav3-python/exllamav3 \
-e VLLM_EXL3_ONLINE_CACHE_DIR=/cache/exl3-online \
docker.io/voipmonitor/vllm:gilded-gnosis-v20-vllmf5981f1-si978cdb3-fi801d57a-cu132-20260803-r25 \
vllm serve /model --served-model-name qwen-3.6 --port 8000 \
--kv-cache-dtype fp8 \
--quantization-config '{"linear":{"weight":"mxfp8"},"shared_experts":{"weight":"mxfp8"},"ignore":["re:.*\\.mlp\\.gate$","re:.*linear_attn.*","re:.*shared_expert_gate.*","re:.*conv1d.*","lm_head","re:.*embed_tokens.*","re:.*visual.*","re:mtp\\..*"]}'
The GDN ignore entries are load-bearing: Qwen's Gated DeltaNet layers
contain micro-projections (in_proj_ba, N=64; shared_expert_gate, N=1)
too narrow for the online-K6 encoder (128-alignment) AND the MXFP8
fallback (mm_mxfp8 needs N>=128) — they must stay BF16 or the engine
fails at memory profiling. Two matching subtleties: fused modules are
ignore-matched against their UNFUSED shard names (so in_proj_ba$ never
matches — use the blanket re:.*linear_attn.*), and shared-expert
projections bypass the ignore list entirely (governed by the
shared_experts spec).
Notes: quantization auto-selects exl3 from the top-level hybrid_tr3_tail
config block; runtime TP must equal the checkpoint's tp: 1; the rank-sliced
path is CUDA-graph-safe (no --enforce-eager needed); do not enable
--speculative-config (the MTP experts are BF16 in this build). First start
builds the online-K6 cache under /cache/exl3-online; warm restarts hit it.
tools/The complete, reviewed toolchain that produced this checkpoint (fork of the
GLM-5.2 calibration_encoder bundle, single-GPU sm_120 edition):
capture_qwen36.py — TP1 BF16 calibration capture (vLLM cpu_offload_gb
streams the 67 GiB source through 32 GiB of VRAM; softmax top-8 router
recompute; 5×8-layer tmpfs windows; 1,052,436-token owner corpus).encode_qwen36.py — adapter over the byte-pinned production encoder
encode_tr3_v31.py (sha256 e9a85a47…75032, LDLQ + MCG Trellis,
exllamav3 0.0.43): fused-3D expert reader, per-expert K via uniform-K
pools, VL-preserving assembly.k_plan_qwen36.py — router-mass-weighted K4 selection (top-N by routed
token count per layer), sealed plan + tier_bitmap.preflight_qwen36.py, bootstrap_ext_sm120.py, convert_qwen36.sh,
run_campaign_qwen36.sh — gates, sm_120 quantizer-ext build, orchestration.patches/0001-qwen3_5-rank-sliced-exl3-loading.patch — the 2-file vLLM-GG
loader patch (rank-slice name normalization for the qwen3_5 family).Pipeline gates: adversarial code review, per-K encoder smoke
(lockstep == sequential byte-identity), --verify-engine router-recompute
audit on captured data, per-K byte-equality oracle, fingerprinted resume at
every stage, byte-exact carried-tensor verification at assembly.
calibration_encoder bundle (capture/encode/bootstrap toolchain + the
owner calibration corpus) this fork derives from.