rebuttal_w3g128_awq_w2

Qwen3-1.7B — AWQ W3/g128 (look-ahead loss), dequantized to BF16

Weights are 3-bit AWQ values stored dequantized in BF16, so this loads with a stock AutoModelForCausalLM.from_pretrained — no AWQ kernels or custom classes. Every decoder-block linear holds at most 8 distinct values per group of 128 input channels; embeddings and lm_head are untouched (weight-only quantization).

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
m = AutoModelForCausalLM.from_pretrained(
    "ghostchoir/rebuttal_w3g128_awq_w2", torch_dtype=torch.bfloat16, device_map="cuda")
tok = AutoTokenizer.from_pretrained("ghostchoir/rebuttal_w3g128_awq_w2")

Quantization

Search objective

The candidate for block i is scored one block later, at the output of block i+1, which stays in original precision:

inps ──► block i (candidate weights) ──► block i+1 (BF16, untouched) ──► MSE

The reference is the same path with block i also unquantized. Block 27 has no successor, so its loss falls back to its own output.

Because AWQ's scale rewrite (prev_op /= s, W *= s) is algebraically equal to leaving prev_op alone and using Q(W·diag(s))·diag(1/s), a candidate is simulated by swapping only the linear's weight, so the measured loss is exactly the deployed one.

Clipping granularity. The look-ahead loss is a single scalar after block i+1 and does not decompose over output channels, so clipping searches one shrink ratio per linear rather than stock AWQ's per-output-channel threshold.

Calibration

Text is tokenized as-is — no chat template applied, no stripping (rows already carry ChatML markup).

Results

Perplexity at sequence length 2048. Bold is this model.

Held-out slice = rows 4096–5119 of the calibration dataset (never seen by the search). RTN = round-to-nearest group quantization with no AWQ search.

Comparability caveat. The d=0 and d=1 arms differ in two ways, not one: the loss scope (block output vs next-block output) and the clipping search (stock per-output-channel vs a single global shrink scored look-ahead). The clipping difference does more work at 3-bit than at 4-bit, so the gap between these two arms should not be read as an ablation of the loss alone.