rebuttal_w3g128_awq

Qwen3-1.7B — AWQ W3/g128 (block-wise loss), dequantized to BF16

Weights are 3-bit AWQ values stored dequantized in BF16, so this loads with a stock AutoModelForCausalLM.from_pretrained — no AWQ kernels or custom classes. Every decoder-block linear holds at most 8 distinct values per group of 128 input channels; embeddings and lm_head are untouched (weight-only quantization).

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
m = AutoModelForCausalLM.from_pretrained(
    "ghostchoir/rebuttal_w3g128_awq", torch_dtype=torch.bfloat16, device_map="cuda")
tok = AutoTokenizer.from_pretrained("ghostchoir/rebuttal_w3g128_awq")

Quantization

Search objective

The candidate is scored by the MSE at the output of the block being quantized:

inps ──► block i (candidate weights) ──► MSE vs block i unquantized

Note this is a block-wise objective. Stock AWQ scores at the output of the sub-module it perturbs (self_attn, mlp, or a single nn.Linear), which is a narrower scope than the whole block. Clipping is stock AWQ's per-output-channel threshold search.

Calibration

Text is tokenized as-is — no chat template applied, no stripping (rows already carry ChatML markup).

Results

Perplexity at sequence length 2048. Bold is this model.

Held-out slice = rows 4096–5119 of the calibration dataset (never seen by the search). RTN = round-to-nearest group quantization with no AWQ search.

Comparability caveat. The d=0 and d=1 arms differ in two ways, not one: the loss scope (block output vs next-block output) and the clipping search (stock per-output-channel vs a single global shrink scored look-ahead). The clipping difference does more work at 3-bit than at 4-bit, so the gap between these two arms should not be read as an ablation of the loss alone.