Downloads · 30 days
21
1% of all-time downloads
rdtand/Qwen3.6-27B-prismaquant-gridbook-5.5bit-vllm
Qwen3.6-27B-prismaquant-gridbook-5.5bit-vllm is a image-text-to-text model from rdtand. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for gridbook. The card lists the license as apache-2.0.
23.0 GB codebook-quantized Qwen3.6-27B (vision-language, MTP included) serving through stock vLLM with the open gridbook plugin — no fork, no core patches. Weights are stored as product-VQ codebook indices whose codeb…
Downloads · 30 days
21
1% of all-time downloads
All-time downloads
2.6K
Public
Parameters
19.7B
23 GB on disk
Likes
6
Public
Click a slice to open those files.
.safetensors23 GB · 100%
How the weights are stored.
U816.4B · 83%
From the Hugging Face model README
23.0 GB codebook-quantized Qwen3.6-27B (vision-language, MTP included) serving through stock vLLM with the open gridbook plugin — no fork, no core patches. Weights are stored as product-VQ codebook indices whose codebook values lie exactly on hardware grids (e4m3 / E2M1), so serving decodes into bit-standard FP8/NVFP4 tensors and runs on the same tensor-core GEMM paths a conventional checkpoint uses.
Protocol: top-20 prompt_logprobs KL on a 12.3k-token held-out corpus
(disjoint from calibration), 8,176 positions, all arms served back-to-back on
one box against the same teacher capture — a strict relative comparison.
| artifact | size | KL (all) | KL (confident half) | top-1 (conf) |
|---|---|---|---|---|
| this artifact (CB ladder, 5.5 bpp body) | 23.0 GB | 0.0049 | 0.00295 | 99.88% |
| Qwen3.6-27B-PrismaAURA-5.5bit (NVFP4+FP8) | 23 GB | 0.0211 | 0.0130 | 99.58% |
| Qwen3.6-27B-PrismaSCOUT 5p31 | 20.6 GB | 0.0344 | 0.0249 | 99.04% |
At matched size and matched body bitrate, the codebook ladder carries −77% KL versus the strongest conventional (NVFP4+FP8) artifact in this family. Same model, same bytes, same protocol — the delta is the format.
| section | format | notes |
|---|---|---|
| 504 body Linears | FP8-CB K36–K47 (426 units) + BF16 (69) | per-Linear rungs chosen by measured error under a byte budget; K47 (5.88 bpw) ×175, K44 ×109, K40 ×108 — the ⅛-bpw all-integer ladder in actual use |
| vision tower (110 Linears) | NVFP4 weight-only (W4A16) | quantized ViT, verified on image inputs |
| MTP draft block | BF16 | canon throughput selector: highest-fidelity rung that fits |
| embeddings + lm_head | BF16 | 248k vocab, untied |
Full per-Linear assignment ships in allocation/. Codebooks (cb_codebooks.pqcb)
are shared per role family and total a few MB.
| Plugin | gridbook — an out-of-tree vLLM quantization plugin. Stock vLLM, no fork, no core patches. |
| GPU | NVIDIA Blackwell, compute capability sm_120 / sm_121. Measured on GB10 / DGX Spark (sm_121). On older GPUs the plugin still loads but runs its Triton fallback kernels — correct, not fast, and not a production serving target. |
| Memory | 20.8 GiB weights + KV cache, measured on one 128 GB unified-memory GB10 (Model loading took 20.82 GiB). Before gridbook 0.1.0 the same artifact loaded at 35.86 GiB — dense codebook weights were resident twice (issue #1, fixed in 0.1.0); use 0.1.0 or newer. A 32 GB consumer Blackwell (RTX 5090, sm_120) should now fit weights plus a useful KV budget, and the issue reporter measured 20.54 GiB with 32k context using equivalent patches — but we have not verified 0.1.0 on a 32 GB card ourselves. |
| vLLM | Served here on the vllm-node image used for the 0.1.0 verification; also exercised on 0.25.1 and 0.23.1rc1.dev1060 by an external reporter (issue #1). The plugin surface is small but vLLM internals drift — pin a version you have tested. |
| Toolchain | CUDA toolkit with nvcc on PATH in the serving container — the plugin JIT-builds its kernels on first model load (~30 s, cached). nvcc 13.0 is the tested toolchain. |
| Parallelism | Single GPU (tp=1). Tensor parallelism is not implemented in the plugin. |
pip install gridbook # JIT-builds kernels for your GPU (capability >= 8.0 floor)
vllm serve rdtand/Qwen3.6-27B-prismaquant-gridbook-5.5bit-vllm \
--max-model-len 32768 --gpu-memory-utilization 0.90
Quantized with PrismaQuant (measurement-driven per-Linear format allocation); format spec and kernels: gridbook, Apache 2.0.
Contact: [email protected]