DHSA-Llama-3.1-8B-Instruct-NF4

DHSA-Llama-3.1-8B-Instruct-NF4

This repository hosts the predictor weights for Llama-3.1-8B-Instruct (NF4) used in Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference (ICML 2026 Spotlight).

Checkpoint Schema

Every checkpoint contains:

Usage

First, clone our repository and download the checkpoint files from Hugging Face:

git clone https://github.com/xiongsiheng/DHSA
cd DHSA

hf download sxiong/DHSA-Llama-3.1-8B-Instruct-NF4 \
  --local-dir checkpoints/DHSA-Llama-3.1-8B-Instruct-NF4

Pass the desired checkpoint to the evaluation command:

CUDA_VISIBLE_DEVICES=0 python run_ruler.py \
  --model_name meta-llama/Llama-3.1-8B-Instruct \
  --use_quant \
  --data_dir data/RULER \
  --sparsity-mask DHSA_learned_topK_static \
  --predictor-checkpoint checkpoints/DHSA-Llama-3.1-8B-Instruct-NF4/predictor_static.pt \
  --density 0.125 \
  --q-block-size 128 \
  --k-block-size 32 \
  --save_dir results_ruler

Citation

@inproceedings{xionglong,
  title={Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference},
  author={Xiong, Siheng and Zou, Joe and Fekri, Faramarz and Cho, Yae Jee},
  booktitle={Forty-third International Conference on Machine Learning}
}