Downloads · 30 days
12
5% of all-time downloads
rdtand/Hy3-295B-A21B-prismaquant-gridbook-2.9bit-vllm
Hy3-295B-A21B-prismaquant-gridbook-2.9bit-vllm is a text generation model from rdtand. Use it when you need the model to write or continue text. It is set up for gridbook. The card lists the license as apache-2.0.
A 295B-A21B MoE served on one 128 GB DGX Spark at 2.9 bits per parameter, through stock vLLM with an out-of-tree plugin — no forked runtime, no core patches. Weights are stored in the gridbook codebook formats (NVFP4-…
Downloads · 30 days
12
5% of all-time downloads
All-time downloads
220
Public
Parameters
104B
106 GB on disk
Likes
3
Public
Click a slice to open those files.
.safetensors106 GB · 100%
How the weights are stored.
U8102B · 98%
From the Hugging Face model README
A 295B-A21B MoE served on one 128 GB DGX Spark at 2.9 bits per parameter, through stock vLLM with an out-of-tree plugin — no forked runtime, no core patches. Weights are stored in the gridbook codebook formats (NVFP4-CB / FP8-CB: product-VQ over 8-weight vectors whose codebook values live on hardware grids, two-tier scale coding) and decoded inside custom CUDA/Triton kernels at matmul time, so prefill runs on native tensor cores instead of paying a dequantization tax. Formats + kernels: github.com/RobTand/gridbook. Produced by the PrismaQuant measured-allocation pipeline.
allocation/mtp_rung_selection.json). Draft quantization can
only affect acceptance rate — rejection sampling reproduces the target
distribution exactly. A GGUF export cannot carry MTP at all.allocation/ (per-Linear assignment, Pareto
sweep, MTP rung selection).| Metric | This artifact | GGUF IQ 2.8 bpp (same box) |
|---|---|---|
| Prefill (1.5k-token prompt) | ~109 tok/s | 42 tok/s |
| Decode (batch 1, base) | 14.6 tok/s | 17.8 tok/s |
| Decode (batch 1, prose, MTP spec on) | 16.1 tok/s | not possible |
The prefill gap is the point of the formats: IQ/k-quant GGUF decodes weights on the fly in CUDA-core kernels; gridbook expands each layer transiently and runs native tensor-core GEMMs, then frees the tile. Base batch-1 decode still trails the GGUF k-quant/MMVQ path (the fp4-CB decode chain is compute-bound at GEMV shapes — measured and documented in the gridbook repo); MTP speculative decode closes most of that gap on natural text and becomes a straight multiplier once vLLM captures drafter CUDA graphs.
Exact on-disk accounting from the shipped shards + quant_config.json (CB
totals include packed cb_qweight and fp8-rung weight_scale sidecars). A
unit is one allocation target: for routed experts, one stacked tensor
holding all 192 experts of one projection — two stacks per MoE layer; otherwise
one Linear. "bits/w" is the rung's body bit-rate (NVFP4-CB two-tier:
k/8 + 0.28125; FP8-CB: k/8 plus per-channel scale; vanilla FP8: 8).
Naming note: "FP8"/"NVFP4" in a CB format names the grid the codebook's VALUES live on, not the storage width. Storage is the k-bit vector index; the reconstructed values are exact e4m3 (or E2M1) numbers — which is why an expanded FP8-CB tile is a standard per-channel fp8 tensor that vLLM's stock fp8 tensor-core GEMM consumes directly. "FP8-CB K32" is a 4.0-bit weight encoding served through native fp8 kernels.
| Format | bits/w | Role | Units | Size (GB) |
|---|---|---|---|---|
| NVFP4-CB K14 | 2.03 | routed experts | 18 stacks | 8.281 |
| NVFP4-CB K14 | 2.03 | shared expert | 25 | 0.040 |
| NVFP4-CB K14 | 2.03 | dense/attention | 16 | 0.082 |
| NVFP4-CB K16 | 2.28 | routed experts | 36 stacks | 18.601 |
| NVFP4-CB K16 | 2.28 | shared expert | 8 | 0.014 |
| NVFP4-CB K16 | 2.28 | dense/attention | 3 | 0.012 |
| NVFP4-CB K18 | 2.53 | routed experts | 38 stacks | 21.786 |
| NVFP4-CB K18 | 2.53 | dense/attention | 2 | 0.035 |
| NVFP4-CB K20 | 2.78 | routed experts | 28 stacks | 17.638 |
| NVFP4-CB K20 | 2.78 | dense/attention | 6 | 0.029 |
| FP8-CB K28 | 3.5+ | routed experts | 38 stacks | 30.228 |
| FP8-CB K28 | 3.5+ | dense/attention | 26 | 0.331 |
| FP8-CB K32 | 4.0+ | dense/attention | 194 | 1.790 |
| FP8-CB K32 | 4.0+ | shared expert | 133 | 0.419 |
| FP8-CB K36 | 4.5+ | dense/attention | 33 | 0.326 |
| FP8-CB K44 | 5.5+ | dense/attention | 23 | 0.289 |
| FP8-CB K44 | 5.5+ | shared expert | 8 | 0.035 |
| FP8-CB K44 | 5.5+ | MTP experts + attention + shared | 9 | 2.562 |
| vanilla FP8 | 8 | dense/attention | 16 | 0.243 |
| vanilla FP8 | 8 | shared expert | 20 | 0.126 |
| BF16 | 16 | embeddings | 1 | 0.990 |
| BF16 | 16 | lm_head | 1 | 0.990 |
| BF16 | 16 | shared-expert Linears (allocator-assigned) | 43 | 0.541 |
| BF16 | 16 | norms (incl. q/k norms) | 325 | 0.152 |
| BF16 | 16 | router gates | 79 | 0.124 |
| BF16 | 16 | MTP glue (eh_proj, enorm/hnorm, norms) | 9 | 0.069 |
| F32 | 32 | expert bias (routing) | 80 | <0.001 |
| Total | 105.732 |
Family rollup: NVFP4-CB 180 units / 66.5 GB · FP8-CB 464 units / 36.0 GB · vanilla FP8 36 units / 0.37 GB · BF16+F32 538 units / 2.9 GB.
Per-layer view of the 79 MoE layers (experts are format-uniform within a layer, mixed across layers — a vLLM fused-MoE serving constraint): 9 layers on NVFP4-CB K14 · 18 on K16 · 19 on K18 · 14 on K20 · 19 on FP8-CB K28. Layer 0 is dense; layer "80" is the MTP module (FP8-CB K44 throughout + BF16 glue).
A 295B cannot be KL-validated against its BF16 teacher on this hardware, so — as with our GGUF Hy3 releases — this card makes no quality claims. The validation ledger: vLLM loads and serves it; generation is coherent and factually/arithmetically correct on our smoke suite; packing is bit-exact against the reference codec (test-pinned); tool-calling behaviour was exercised end-to-end under the hardmode harness we run on every release. Quantization is calibrated (activation-weighted codebook encode; measured per-Linear format allocation on real calibration signals; at validatable scales — 27B/35B — the same formats measured −53 to −58% served KL against same-size conventional artifacts).
| Plugin | gridbook — an out-of-tree vLLM quantization plugin. Stock vLLM, no fork, no core patches. |
| GPU | NVIDIA Blackwell, compute capability sm_120 / sm_121. Measured on GB10 / DGX Spark (sm_121). On older GPUs the plugin still loads but runs its Triton fallback kernels — correct, not fast, and not a production serving target. |
| Memory | ~106 GB resident weights (including the CB-quantized MTP drafter) on a ~121 GB usable unified pool — one 128 GB DGX Spark. --gpu-memory-utilization 0.90 is the validated setting; higher values OOM'd under long-prefill activation spikes. |
| vLLM | ≥ 0.23 with HYV3ForCausalLM (stock — no fork). Speed numbers on this card were measured on vLLM 0.23. |
| Toolchain | CUDA toolkit with nvcc on PATH in the serving container — the plugin JIT-builds its kernels on first model load (~30 s, cached). nvcc 13.0 is the tested toolchain. |
| Parallelism | Single GPU (tp=1). Tensor parallelism is not implemented in the plugin. |
Requirements: vLLM >= 0.23 with HYV3ForCausalLM (stock), CUDA toolkit with
nvcc (the plugin JIT-builds its kernels on first load), ~121 GB usable
GPU/unified memory.
pip install gridbook
MODEL_DIR=. ./serving/serve.sh # 0.0.0.0:8000, 12k ctx, fp8 KV, MTP on
The gridbook plugin (serving/gridbook/, Apache 2.0 — canonical home
github.com/RobTand/gridbook) registers the gridbook quantization method
through vLLM's standard plugin entry point and serves the codebook formats
with:
PRISMAQUANT_CB_DECODE=triton).Memory: serve at --gpu-memory-utilization 0.90 (validated; higher settings
OOM under long-prefill activation spikes on a 128 GB Spark). Context 12288
validated; fp8 KV recommended.
gridbook formats + kernels and PrismaQuant allocation pipeline —
Robert Tand ([email protected]).
Base model by Tencent (Apache 2.0; see LICENSE).