HGRN-1.3B-semi-structured-EHWS-1phase-80pct-old

HGRN-1.3B, 80% semi-structured sparsity (EHWS, one phase, older model)

This is fla-hub/hgrn-1.3B-100B pruned to 80% sparsity with an older, one-phase version of EHWS.

This is an older model. Our current HGRN-1.3B model at 80% is HGRN-1.3B-semi-structured-EHWS-80pct (40.31 / 29.82).

Every row of every pruned linear layer keeps the same number of weights (the top 20%). Embeddings and lm_head are not pruned. The zeros are stored in the weights.

How to load

You need flash-linear-attention installed.

import fla
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ikimyaii/HGRN-1.3B-semi-structured-EHWS-1phase-80pct-old"
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo)

Method

This version skips Phase 1 (--skip-phase1). It runs one global ADMM run on the whole model, starting from the dense weights.

• Loss: 0.5 x CE + 0.5 x KD (the dense model is the teacher)
• 128 rounds x 32 steps, batch 8
• lr 6e-4, lambda 1e-2, Adam beta2 0.95
• bf16
• 2,048 C4 sequences of 2,048 tokens

The run took about 6.6 hours on one A100 40GB.

Results

Perplexity uses ELSA's evaluation code (WikiText-2 and C4, fp16). One run.

Measured sparsity: 79.99%. Every row has the same number of nonzeros (checked on these weights).

Zero-shot accuracy (lm-evaluation-harness, 0-shot):

Code

github.com/KimiaGholami/EHWS (run_pipeline.py --skip-phase1)