hgrn-1.3B-cws-80pct

HGRN-1.3B · CWS 80% sparse

Renamed from hgrn-1.3B-obs-cancel-block-80pct on 2026-10-13 as part of the OBS-cancel → CWS rename; the old URL redirects here.

HGRN-1.3B, a 1.3B-parameter gated linear-recurrent language model with no attention, pruned one-shot to 80% unstructured weight sparsity with CWS. No retraining was done after pruning.

Model details

What is pruned: every nn.Linear in all 24 HGRN blocks (recurrent projections i_proj, f_proj, g_proj, o_proj and MLP gate_proj, up_proj, down_proj) plus lm_head. Embeddings and norms are dense. Per-layer counts are in pruning_summary.json.

Provenance note: this repo was originally named …obs-cancel-block…, but its pruning_summary.json records "method": "obs_cancel", the global selection scope, with C4 calibration. The card follows the recorded metadata. Its weights are byte-identical to hgrn-1.3B-cws-global-80pct.

Evaluation

At 80% sparsity every method collapses to near-chance accuracy (LAMBADA → 0). This checkpoint keeps the lowest one-shot PPL among the tested methods: SparseGPT 6,956, AWP 17,195, RIA 28,615, Wanda 76,051.

Evaluation protocol: WikiText-2 test perplexity uses non-overlapping 2048-token windows. Zero-shot tasks use lm-evaluation-harness, reporting acc_norm for ARC-e/ARC-c/HellaSwag/PIQA and acc for WinoGrande/LAMBADA. The dense baseline is ikimyaii/hgrn-1.3B-dense-baseline (fla-hub/hgrn-1.3B-100B). Raw results: results/benchmark_hgrn_obs_cancel_block_80pct.json.

At 80% sparsity the model does not produce usable text. It is published for hardware sparsity and efficiency benchmarking, and as a starting point for recovery training (see the -ft and -distill variants).

About CWS

CWS (Correlation-aware Weight Sparsity) is a post-training, one-shot pruning method that extends SparseGPT's Optimal Brain Surgeon (OBS) weight correction with a full-Hessian selection criterion.

Per-weight saliency scores (magnitude, Wanda, RIA, SparseGPT's diagonal score) look at each weight in isolation. They miss cancellation groups: correlated input channels whose opposite-sign weights partly cancel in the layer output, so removing them together costs less than their individual scores predict. For each output row, CWS greedily picks the weight with the smallest marginal cost δ(j | S) = r_j² / d_j, where r_j and d_j come from the evolving inverse-Hessian factor (one rank-1 Schur-complement downdate per pick). It then applies exact OBS corrections to the surviving weights. With H = (2/N) XᵀX, the first pick is exactly SparseGPT's choice; later picks account for the cross-weight interactions that a diagonal score ignores.

Two selection scopes appear in this collection:

• Block CWS (default): greedy selection is confined to 128-column blocks, and corrections are applied block by block. This is stable on wide layers.

• Global CWS (ablation): the whole row is one block (block size → ∞). It captures cross-block cancellation, but it is more prone to numerical drift on large layers.

• Method and reference implementation: CWS

• Experiments, baselines and raw results for these checkpoints: Sparse-LLMs-on-Loihi

• Follow-up work that swaps CWS's layer-wise reconstruction objective for the real next-token loss: Weight-Sparsity-Using-Real-Loss

Usage

The model uses the HGRN architecture from flash-linear-attention (FLA).

pip install flash-linear-attention transformers torch

import torch
import fla  # registers HGRN with transformers
from fla.models.hgrn import HGRNConfig, HGRNForCausalLM
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

AutoConfig.register("hgrn", HGRNConfig, exist_ok=True)
AutoModelForCausalLM.register(HGRNConfig, HGRNForCausalLM, exist_ok=True)

model_id = "ikimyaii/hgrn-1.3B-cws-80pct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).cuda()

# Pruned weights are stored as explicit zeros in a dense tensor.
lin = model.model.layers[0].mlp.down_proj
print(f"sparsity: {(lin.weight == 0).float().mean():.3f}")

The checkpoint is stored densely, with zeros in the pruned positions, so it runs as-is on GPUs and CPUs. Speedups require a sparse kernel or hardware that skips zero weights. On Intel Loihi and other event-driven neuromorphic hardware, a zero weight generates no synaptic event, so the sparsity directly reduces active computation.

Limitations

• These are research checkpoints for studying pruning. They have not been instruction-tuned or safety-tuned, and they inherit the biases of the base model and its pre-training data.
• At 80% sparsity, outputs are largely incoherent; zero-shot accuracy is near chance.
• The weights are stored densely. Sparsity alone gives no speedup without a sparse runtime or neuromorphic hardware.

Citation

The CWS paper is under anonymous review; please cite the code repository for now.

@misc{cws2026,
  title  = {Semi-Structured Correlation-aware Weight Sparsity (CWS) for Efficient LLM Inference},
  year   = {2026},
  note   = {Under review. Code: https://github.com/KimiaGholami/CWS}
}

@inproceedings{frantar2023sparsegpt,
  title     = {SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot},
  author    = {Frantar, Elias and Alistarh, Dan},
  booktitle = {International Conference on Machine Learning},
  year      = {2023}
}