llama_3.1_8b-PATCH-35Sparse

<div align="center"> <img src="./PATCH-Logo.png" alt="PATCH" width="360"> </div>

llama_3.1_8b-PATCH-35Sparse

This checkpoint (LLaMA-3.1 8B, PATCH-Tile, 35% sparsity): 55.28% average zero-shot accuracy, 7.89 WikiText2 perplexity.

This repository hosts a mask only release for the paper PATCH: Learnable Tile-level Hybrid Sparsity for LLMs. PATCH (Pruning with a Learnable Tile-level Configuration for Hybrid Sparsity) learns a structured mask on frozen pretrained weights, assigning each tile as dense (0% sparsity) or 2:4 sparse (50% sparsity) to hit a flexible global sparsity target while staying hardware-friendly.

Because PATCH/MaskLLM keep the base weights frozen, we distribute only the binary keep/prune mask (bit-packed in mask.npz) - no weight values. You recover the sparse model by downloading the original base model and applying the mask.

• Base model: meta-llama/Llama-3.1-8B
• Method: PATCH-Tile  |  Target sparsity: 35%  |  Pattern: Dense / 2:4 tiles
• Measured mask sparsity: 35.00% over 224 Linear layers (6,979,321,856 weights).

<div align="center"> <img src="./PATCH-Pipeline.svg" alt="PATCH pipeline" width="760"> </div>

Results (LLaMA-3.1 8B)

Per-task zero-shot accuracy (%) for this checkpoint:

All numbers are from the PATCH paper (arXiv:2509.23410); accuracy is the average over MMLU, PIQA, ARC-Easy, ARC-Challenge, Winogrande, OpenBookQA, RACE and HellaSwag, evaluated with the LM-Evaluation-Harness. PPL is WikiText2.

Training hyper-parameters

How to use

from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
import torch
from load_patch_mask import apply_patch_mask  # shipped in this repo

npz = hf_hub_download(repo_id="mohammad-mozaffari/llama_3.1_8b-PATCH-35Sparse", filename="mask.npz")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B", torch_dtype=torch.bfloat16)
apply_patch_mask(model, npz)   # zeroes the pruned weights in place

Or from the command line:

python load_patch_mask.py --base_model meta-llama/Llama-3.1-8B --mask_repo mohammad-mozaffari/llama_3.1_8b-PATCH-35Sparse

Speedup on real hardware requires a 2:4-aware / hybrid sparse kernel; see the GitHub repository and STOICC.

License

The released mask is a derivative of the base model and is distributed under the base model's license (llama3.1). You must comply with that license and obtain access to the base model separately.

Built with Llama. Use governed by the Llama 3.1 Community License.

The mask-generation code is released under the MIT license (see the PATCH repository).

Citation

@article{hourri2025patch,
    title  = {PATCH: Learnable Tile-level Hybrid Sparsity for LLMs},
    author = {Hourri, Younes and Mozaffari, Mohammad and Mehri Dehnavi, Maryam},
    year   = 2025,
    journal = {arXiv preprint arXiv:2509.23410}
}