Qwen3-8B-BPDQ-2bit

Qwen3-8B — 2-bit BPDQ (packed)

Qwen3-8B, all 8.2B parameters, with the transformer weights stored as 2-bit bit-planes and kept that way for the whole run. 3.0 GB on disk, 3.65 GiB resident, against 15.3 GiB in fp16. Runs on one CUDA GPU or an Apple M-series Mac. (The Hub's "0.8B params" badge counts packed int32 words, not parameters.)

Benchmarks

12-task average 72.0 against 80.6 for fp16, with the weights at 2.19 bits. At the same bit budget llama.cpp's imatrix g256 quant scores 17.8; it needs 3.25 bits/weight (74.1) to pass this model.

On one RTX 4090 it solves 5.4x more GSM8K problems per minute than fp16 for a single user and 2.6x at 64 concurrent users; the gap narrows as batching amortizes the weight read.

All rows measured 2026-10-09 to 10-11 on the same harness as the fp16 reference.

Run it

import sys; sys.path.insert(0, MODEL_DIR)
import bpdq

model, tok = bpdq.load(MODEL_DIR)                      # cuda | mps | cpu
ids = tok("The capital of France is", return_tensors="pt").to(model.device)
out, _ = bpdq.decode(model, ids.input_ids, 64, sampling=bpdq.Sampling.for_mode(think=False))
print(tok.decode(out[0]))

python serve.py MODEL_DIR --port 8000      # OpenAI-compatible, streaming, standard library only
python bpdq.py chat MODEL_DIR              # interactive

On CUDA, import bpdq registers the method with vLLM: LLM(model=MODEL_DIR, quantization="bpdq", dtype="float16") or vllm serve MODEL_DIR --quantization bpdq --dtype float16. demo.ipynb walks through all of it on either machine.

Sampling defaults are Qwen3's per-mode settings plus a 0.3 presence penalty; do not decode greedily, at 2 bits it restates itself. Everything else — the weight format, the kernels, per-batch throughput, speculative decoding, server flags — is in ENGINEERING.md.

Quality

Limits

• TP = 1. No tensor parallelism.
• On Apple, batches of 64 tokens and up run at 0.7–0.9x of a dense fp16 matmul. The win is at decode batch sizes.
• At 2 bits the model invents confident detail about anything obscure, and past roughly 250 tokens long open-ended answers drift. Use it for short exchanges and bounded tasks.
• The torch decode loop does not capture CUDA graphs; transformers bakes part of the attention mask into a capture and the replayed step drifts. vLLM's own capture is fine.
• Greedy output is not bit-stable across cache implementations — this loop uses a StaticCache, model.generate a dynamic one. Both are faithful to the weights.
• Launch shapes were measured on an RTX 4090 and an M4; other chips run correctly but may not be at their own optimum.

License

Apache-2.0, inheriting Qwen/Qwen3-8B.