rebuttal_w4g128_cbq

Qwen3-1.7B — CBQ W4G128

Qwen/Qwen3-1.7B quantized to W4G128 with CBQ (Cross-Block Quantization, arXiv:2312.07950), then dequantized back to bf16.

This is a dense bf16 checkpoint, not a packed low-bit one. Every weight has been round-tripped through the W4G128 grid, so it carries the accuracy of W4G128 quantization, but the file itself is an ordinary Qwen3ForCausalLM in bfloat16 with the same tensor names and shapes as the base model. It loads in vLLM or transformers with no custom kernel, no quantization_config, and no code change — and it gives no speed or memory benefit over the base model. Use it to evaluate what W4G128 costs in quality.

from vllm import LLM
llm = LLM("ghostchoir/rebuttal_w4g128_cbq")            # dtype is bfloat16 per config.json

Configuration

Calibration: 4096 rows of open-r1/OpenR1-Math-220k rendered with the Qwen3 chat template and truncated to 2048 tokens. This is not the paper's calibration set (the paper uses 128 samples); it is matched to a sibling QAT framework so the two are comparable.

Perplexity above is on held-out rows of the same distribution and is a sanity metric only — it is not a reasoning-benchmark score.

Provenance and caveats

Produced by an unofficial CBQ implementation written from the paper text; CBQ has no official code release. The paper is ambiguous or self-inconsistent in several places (notably the interaction of Eq. 8 and Eq. 11 for Delta_W initialisation, and the direction of the Algorithm 1 objective), and every such choice is documented with the measurement behind it. Two settings differ from the paper deliberately: group-wise rather than per-channel quantization where noted, and the calibration set above.

This checkpoint uses --cfp_weight off and --hard_round_frac 1.0 rather than the paper's values. Both paper settings were measured worse at group-128: CFP weight truncation costs ~+2.6 perplexity because at group-128 an outlier only inflates its own 128-wide group, and rounding Delta_W only in the late phase leaves most of the optimization on a near-lossless (degenerate) objective.

Not affiliated with the CBQ authors or with Qwen. Inherits the base model's license.