gemma_3_1b-MaskLLM-50Sparse

<div align="center"> <img src="./PATCH-Logo.png" alt="PATCH" width="360"> </div>

gemma_3_1b-MaskLLM-50Sparse

This checkpoint (Gemma-3 1B, MaskLLM, 50% sparsity): 41.84% average zero-shot accuracy, 12.82 WikiText2 perplexity.

This repository hosts a mask only release for the paper PATCH: Learnable Tile-level Hybrid Sparsity for LLMs. This is a MaskLLM 2:4 baseline, trained with our re-implementation of MaskLLM (learnable semi-structured 2:4 sparsity via Gumbel-Softmax over frozen weights). It is released as a baseline for the PATCH paper.

Because PATCH/MaskLLM keep the base weights frozen, we distribute only the binary keep/prune mask (bit-packed in mask.npz) - no weight values. You recover the sparse model by downloading the original base model and applying the mask.

• Base model: google/gemma-3-1b-pt
• Method: MaskLLM  |  Target sparsity: 50%  |  Pattern: 2:4
• Measured mask sparsity: 50.00%.

<div align="center"> <img src="./PATCH-Pipeline.svg" alt="PATCH pipeline" width="760"> </div>

Results (Gemma-3 1B)

Per-task zero-shot accuracy (%) for this checkpoint:

All numbers are from the PATCH paper (arXiv:2509.23410); accuracy is the average over MMLU, PIQA, ARC-Easy, ARC-Challenge, Winogrande, OpenBookQA, RACE and HellaSwag, evaluated with the LM-Evaluation-Harness. PPL is WikiText2.

Training hyper-parameters

How to use

from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
import torch
from load_patch_mask import apply_patch_mask  # shipped in this repo

npz = hf_hub_download(repo_id="mohammad-mozaffari/gemma_3_1b-MaskLLM-50Sparse", filename="mask.npz")
model = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-pt", torch_dtype=torch.bfloat16)
apply_patch_mask(model, npz)   # zeroes the pruned weights in place

Or from the command line:

python load_patch_mask.py --base_model google/gemma-3-1b-pt --mask_repo mohammad-mozaffari/gemma_3_1b-MaskLLM-50Sparse

Speedup on real hardware requires a 2:4-aware / hybrid sparse kernel; see the GitHub repository and STOICC.

License

The released mask is a derivative of the base model and is distributed under the base model's license (gemma). You must comply with that license and obtain access to the base model separately.

Gemma is provided under and subject to the Gemma Terms of Use.

The mask-generation code is released under the MIT license (see the PATCH repository).

Citation

@article{hourri2025patch,
    title  = {PATCH: Learnable Tile-level Hybrid Sparsity for LLMs},
    author = {Hourri, Younes and Mozaffari, Mohammad and Mehri Dehnavi, Maryam},
    year   = 2025,
    journal = {arXiv preprint arXiv:2509.23410}
}

This checkpoint is a MaskLLM baseline; please also cite MaskLLM:

@inproceedings{fang2024maskllm,
    title     = {MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models},
    author    = {Fang, Gongfan and Yin, Hongxu and Muralidharan, Saurav and Heinrich, Greg and Pool, Jeff and Kautz, Jan and Molchanov, Pavlo and Wang, Xinchao},
    booktitle = {NeurIPS},
    year      = {2024}
}