qwen2.5_0.5b-PATCH-45Sparse

<div align="center"> <img src="./PATCH-Logo.png" alt="PATCH" width="360"> </div>

qwen2.5_0.5b-PATCH-45Sparse

This checkpoint (Qwen-2.5 0.5B, PATCH-Joint, 45% sparsity): 40.29% average zero-shot accuracy, 14.57 WikiText2 perplexity.

This repository hosts a mask only release for the paper PATCH: Learnable Tile-level Hybrid Sparsity for LLMs. PATCH (Pruning with a Learnable Tile-level Configuration for Hybrid Sparsity) learns a structured mask on frozen pretrained weights, assigning each tile as dense (0% sparsity) or 2:4 sparse (50% sparsity) to hit a flexible global sparsity target while staying hardware-friendly.

Because PATCH/MaskLLM keep the base weights frozen, we distribute only the binary keep/prune mask (bit-packed in mask.npz) - no weight values. You recover the sparse model by downloading the original base model and applying the mask.

• Base model: Qwen/Qwen2.5-0.5B
• Method: PATCH-Joint  |  Target sparsity: 45%  |  Pattern: Dense / 2:4 tiles
• Measured mask sparsity: 44.99%.

<div align="center"> <img src="./PATCH-Pipeline.svg" alt="PATCH pipeline" width="760"> </div>

Results (Qwen-2.5 0.5B)

Per-task zero-shot accuracy (%) for this checkpoint:

All numbers are from the PATCH paper (arXiv:2509.23410); accuracy is the average over MMLU, PIQA, ARC-Easy, ARC-Challenge, Winogrande, OpenBookQA, RACE and HellaSwag, evaluated with the LM-Evaluation-Harness. PPL is WikiText2.

Training hyper-parameters

How to use

from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
import torch
from load_patch_mask import apply_patch_mask  # shipped in this repo

npz = hf_hub_download(repo_id="mohammad-mozaffari/qwen2.5_0.5b-PATCH-45Sparse", filename="mask.npz")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B", torch_dtype=torch.bfloat16)
apply_patch_mask(model, npz)   # zeroes the pruned weights in place

Or from the command line:

python load_patch_mask.py --base_model Qwen/Qwen2.5-0.5B --mask_repo mohammad-mozaffari/qwen2.5_0.5b-PATCH-45Sparse

Speedup on real hardware requires a 2:4-aware / hybrid sparse kernel; see the GitHub repository and STOICC.

License

The released mask is a derivative of the base model and is distributed under the base model's license (apache-2.0). You must comply with that license and obtain access to the base model separately.

The mask-generation code is released under the MIT license (see the PATCH repository).

Citation

@article{hourri2025patch,
    title  = {PATCH: Learnable Tile-level Hybrid Sparsity for LLMs},
    author = {Hourri, Younes and Mozaffari, Mohammad and Mehri Dehnavi, Maryam},
    year   = 2025,
    journal = {arXiv preprint arXiv:2509.23410}
}