Downloads · 30 days
10
8% of all-time downloads
ikimyaii/hgrn-1.3B-80pct-distill-200k
hgrn-1.3B-80pct-distill-200k is a text generation model from ikimyaii. Use it when you need the model to write or continue text. It is set up for fla. The card lists the license as mit.
The global-CWS 80% checkpoint after a long (200k-step) mask-fixed distillation run against the dense model. It is the long-schedule counterpart of hgrn-1.3B-cws-global-80pct-distill (50k steps).
Downloads · 30 days
10
8% of all-time downloads
All-time downloads
126
Public
Parameters
1.4B
2.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2.7 GB · 100%
From the Hugging Face model README
The global-CWS 80% checkpoint after a long (200k-step) mask-fixed distillation run against the dense model. It is the long-schedule counterpart of hgrn-1.3B-cws-global-80pct-distill (50k steps).
Provenance: no metadata file was uploaded with these weights. The student and settings follow the 200,000-step configuration in
scripts/distill_sparse.pyand the repo name; they are not confirmed by a run record.
| Student (init) | hgrn-1.3B-cws-global-80pct (formerly hgrn-1.3B-obs_cancel-80pct) |
| Teacher | ikimyaii/hgrn-1.3B-dense-baseline (frozen) |
| Loss | 0.1 × CE + 0.9 × KL(student ‖ teacher) × T², T = 2.0 |
| Data | C4, streaming |
| Schedule | 200,000 steps, LR 2e-5 (scripts/distill_sparse.py) |
| Mask | fixed (pruned weights stay zero) |
| Metric | This model | Dense |
|---|---|---|
| ARC-Easy (acc_norm) | 0.286 | 0.510 |
| ARC-Challenge (acc_norm) | 0.259 | 0.275 |
| HellaSwag (acc_norm) | 0.265 | 0.480 |
| PIQA (acc_norm) | 0.526 | 0.712 |
| WinoGrande (acc) | 0.521 | 0.528 |
| LAMBADA-OpenAI (acc) | 0.001 | 0.383 |
| Average (6 tasks) | 0.310 | 0.481 |
Zero-shot tasks use lm-evaluation-harness. ARC-e, ARC-c, HellaSwag and PIQA report acc_norm; WinoGrande and LAMBADA report acc. The dense baseline is ikimyaii/hgrn-1.3B-dense-baseline, a copy of fla-hub/hgrn-1.3B-100B. Raw results: results/benchmark_hgrn_80pct_distill_200k.json. No PPL was recorded for this run.
Average 0.310, against 0.308 for the 50k-step run and 0.301 one-shot. At 80% sparsity task accuracy stays near chance however long distillation runs.
CWS (Correlation-aware Weight Sparsity) is a post-training, one-shot pruning method that extends SparseGPT's Optimal Brain Surgeon (OBS) weight correction with a full-Hessian selection criterion.
Per-weight saliency scores (magnitude, Wanda, RIA, SparseGPT's diagonal score)
look at each weight in isolation. They miss cancellation groups: correlated
input channels whose opposite-sign weights partly cancel in the layer output,
so removing them together costs less than their individual scores predict.
For each output row, CWS greedily picks the weight with the smallest marginal
cost δ(j | S) = r_j² / d_j, where r_j and d_j come from the evolving
inverse-Hessian factor (one rank-1 Schur-complement downdate per pick). It then
applies exact OBS corrections to the surviving weights. With H = (2/N) XᵀX,
the first pick is exactly SparseGPT's choice; later picks account for the
cross-weight interactions that a diagonal score ignores.
Two selection scopes appear in this collection:
Block CWS (default): greedy selection is confined to 128-column blocks, and corrections are applied block by block. This is stable on wide layers.
Global CWS (ablation): the whole row is one block (block size → ∞). It captures cross-block cancellation, but it is more prone to numerical drift on large layers.
Method and reference implementation: CWS
Experiments, baselines and raw results for these checkpoints: Sparse-LLMs-on-Loihi
Follow-up work that swaps CWS's layer-wise reconstruction objective for the real next-token loss: Weight-Sparsity-Using-Real-Loss
The model uses the HGRN architecture from flash-linear-attention (FLA).
pip install flash-linear-attention transformers torch
import torch
import fla # registers HGRN with transformers
from fla.models.hgrn import HGRNConfig, HGRNForCausalLM
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
AutoConfig.register("hgrn", HGRNConfig, exist_ok=True)
AutoModelForCausalLM.register(HGRNConfig, HGRNForCausalLM, exist_ok=True)
model_id = "ikimyaii/hgrn-1.3B-80pct-distill-200k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).cuda()
# Pruned weights are stored as explicit zeros in a dense tensor.
lin = model.model.layers[0].mlp.down_proj
print(f"sparsity: {(lin.weight == 0).float().mean():.3f}")
The checkpoint is stored densely, with zeros in the pruned positions, so it runs as-is on GPUs and CPUs. Speedups require a sparse kernel or hardware that skips zero weights. On Intel Loihi and other event-driven neuromorphic hardware, a zero weight generates no synaptic event, so the sparsity directly reduces active computation.
The CWS paper is under anonymous review; please cite the code repository for now.
@misc{cws2026,
title = {Semi-Structured Correlation-aware Weight Sparsity (CWS) for Efficient LLM Inference},
year = {2026},
note = {Under review. Code: https://github.com/KimiaGholami/CWS}
}
@inproceedings{frantar2023sparsegpt,
title = {SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot},
author = {Frantar, Elias and Alistarh, Dan},
booktitle = {International Conference on Machine Learning},
year = {2023}
}