24B0908-Week02-Track1-40-Submission01

Week02 / Track 1 (target: 40%) / Submission 01 — SparseGPT

Enrollment No.: 24B0908 Base model: Qwen/Qwen3-4B-Instruct-2507

Method

One-shot, Hessian-aware unstructured weight pruning (Frantar & Alistarh, "SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot", ICML 2023), implemented directly from the reference source (https://github.com/IST-DASLab/sparsegpt) in sparsegpt_core.py. Each linear layer is pruned column-block by column-block using a calibration Hessian built from 128 C4 samples (H = 2 X^T X). Weights are scored by w^2 / [H^-1]_ii^2 and the removed weight's error is propagated to the weights still to be pruned via the inverse Hessian, so the surviving weights absorb the reconstruction error.

• convert_from_hf_checkpoint.py: loads the base model, prunes it to 50% unstructured sparsity across attention and MLP, and writes a self-contained compressed checkpoint (base config, tokenizer, and packed sparse weights).
• dequantize_to_bf16.py: loads that checkpoint and reconstructs a standard dense bf16 HF model directory.
• sparsegpt_core.py: the per-layer pruning algorithm.

Compression ratio

Pruned linears are stored as a packed 1-bit mask plus bf16 values for the surviving weights; everything else (embedding, norms) stays dense bf16. That puts pruned tensors at roughly 9 bits/weight (1 mask bit + 8 value bits), and the embedding table (~10% of parameters, never pruned) adds its full 16 bits on top, which is why the ratio lands at 60% rather than the nominal 50%.

70% sparsity gets the size down to ~42% of the original, closer to the 40% target, but the model at that sparsity collapses into repeating loops with no coherent answer — 0% on both benchmarks. 50% sparsity is the highest we tested that still produces a working model, so that's what this submission uses.

Accuracy (zero-shot, greedy decoding, max_new_tokens=4096)

n=50 per benchmark. Full predictions in results/.

A real drop from baseline. The responses at this sparsity stay coherent and on-topic — the model just answers wrong more often. It never falls into the repetition loops that show up at 70%. During development we checked the pruned weights against the Hessian-based score to confirm the selection is real (it disagrees with plain magnitude pruning on about 10% of weights on a sampled layer), and confirmed the surviving weights are numerically adjusted by the error-compensation step. Details in sparsegpt_submission/DESIGN.md at the project root.

CUDA speedup potential

Unstructured sparsity doesn't speed up a standard dense GEMM kernel by itself — the packed mask shrinks the checkpoint on disk, but this submission decompresses to dense bf16 before running inference. The path to an actual CUDA speedup is NVIDIA's 2:4 semi-structured sparsity on Ampere and newer tensor cores (torch.sparse.SparseSemiStructuredTensor). SparseGPT supports this natively — sparsegpt_core.fasterprune already takes prunen=2, prunem=4 for exactly that pattern — this submission just doesn't use it, since a 2:4 mask is coarser than the calibration-driven unstructured mask used here and would cost some of the accuracy above.

Reproduce

python convert_from_hf_checkpoint.py \
  --model Qwen/Qwen3-4B-Instruct-2507 \
  --output checkpoint_sparsegpt

python dequantize_to_bf16.py \
  --checkpoint checkpoint_sparsegpt \
  --output restored_bf16