24B0908-Week03-Track1-40-Submission01

Week02 / Track 1 (target: 40%) / Submission 02 — Mixed-precision GPTQ

Enrollment No.: 24B0908 Base model: Qwen/Qwen3-4B-Instruct-2507

Method

GPTQ weight quantization (Frantar et al.), applied only to the MLP linears — attention stays at bf16. Same calibration-Hessian error-compensation as standard GPTQ, scoped through GPTQConfig.modules_in_block_to_quantize so only mlp.gate_proj, mlp.up_proj, mlp.down_proj are quantized. MLP is about two-thirds of the model's parameters, so it's the component worth compressing; attention is small enough, and sensitive enough, that leaving it alone costs little size and buys real accuracy back.

• convert_from_hf_checkpoint.py: loads the base model, quantizes the MLP layers to int3 (group size 128), saves a standard GPTQ HF checkpoint.
• dequantize_to_bf16.py: loads that checkpoint, dequantizes the MLP layers to bf16, and restores the attention layers from the base model.

Why the attention layers are restored from the base model, not from the checkpoint

gptqmodel's loader decides which modules are quantized from a fixed list for the model architecture — it doesn't read modules_in_block_to_quantize at all on reload (checked directly: that field never appears anywhere in the installed gptqmodel package). Loading a scoped checkpoint straight through GPTQModel.load() zeroes out every module that was deliberately left at bf16, because the loader tries to treat it as quantized and finds nothing there. dequantize_to_bf16.py works around this: it reads which modules were actually quantized from the checkpoint's own config.json, and for everything else, copies the real weights back in from the base model after gptqmodel's reload finishes. Verified against the base model's weights directly — the restored attention layers match to 0.0 max absolute difference.

Compression ratio

Quantizing only the MLP still gets most of the way to the target, since MLP holds most of the parameters — attention and the unquantized embedding table together are about a third of the model and stay at full precision.

Accuracy (zero-shot, greedy decoding, max_new_tokens=4096)

n=100 per benchmark. Full predictions in results/.

Quantizing the whole model uniformly to int3 at this group size collapses to 19% / 45% — the model loses coherence and loops instead of answering. Leaving attention at bf16 and only quantizing the MLP keeps the model well above that floor at the same bit width. The two errors compound when both components are quantized together; keeping one at full precision gives the model enough headroom to stay coherent. Full sweep this was chosen from is in evaluation/results/gptq_sweep/ANALYSIS.md at the project root.

CUDA speedup potential

int3 GPTQ weights use packed integer storage with per-group scales, which gptqmodel's Triton and ExLlama kernels read directly on CUDA without unpacking to bf16 first — the memory traffic per weight drops close to the bit width, and matmuls become memory-bound on a smaller footprint, which is where GPTQ's actual inference speedup comes from on Ampere-class GPUs and newer. Restricting quantization to the MLP means only the MLP matmuls get that speedup; attention runs at full bf16 speed, no different from an unquantized model. That's a smaller speedup than quantizing everything, in exchange for the accuracy this submission keeps.

Reproduce

python convert_from_hf_checkpoint.py \
  --model Qwen/Qwen3-4B-Instruct-2507 \
  --output checkpoint_mixed_precision

python dequantize_to_bf16.py \
  --checkpoint checkpoint_mixed_precision \
  --output restored_bf16