PSmolLM2-135M

Magnitude Pruning of SmolLM2-135M: A 5% Sparsity Baseline

Abstract

This report investigates weight sparsity in the pretrained SmolLM2-135M language model using one-shot global magnitude pruning.

The objective is to determine how much parameter sparsity can be introduced without substantial degradation in language-model perplexity. The pruning procedure ranks eligible parameters by their absolute magnitude and sets the smallest values to zero. No architectural changes and no post-pruning fine-tuning are applied.

The original model contains 134,515,008 parameters. A 5% target sparsity results in 6,762,828 parameter values being set to zero, corresponding to an actual sparsity of 5.0276%. On a fixed 50,000-token subset of the WikiText-2 test split, perplexity increases from 13.342135 for the original model to 13.401889 after pruning, a relative increase of approximately 0.448%.

The results provide a simple baseline for studying parameter redundancy in pretrained language models and motivate further work on structured, iterative, and activation-aware sparsification.

---

1. Introduction

Modern language models contain millions or billions of learned parameters. An important question in model compression is whether every learned parameter is equally important for the model's final behavior.

One approach to this problem is pruning: selected parameters are removed from active computation, commonly by setting their values to zero.

This report studies a particularly simple pruning strategy, global magnitude pruning. The method assumes that parameters with smaller absolute values are, on average, less important than parameters with larger absolute values.

The goal of this experiment is not to assume a specific compression target such as 50M parameters. Instead, the objective is to empirically measure the relationship between increasing sparsity and model perplexity.

The central question is:

$$ \text{How much sparsity can be introduced before perplexity changes substantially?} $$

---

2. Base Model

The experiments use the pretrained:

SmolLM2-135M

The original checkpoint contains:

$$ 134,515,008 $$

parameters.

The model weights are stored in the original safetensors representation.

For pruning, floating-point tensors with dimensionality of at least two were considered eligible. One-dimensional parameters were preserved.

This resulted in:

$$ 134,479,872 $$

prunable parameters and:

$$ 35,136 $$

protected parameters.

The original checkpoint contained only 19 exact zero-valued parameters.

---

3. Pruning Method

3.1 Global magnitude pruning

For each eligible parameter (w_i), its magnitude is calculated as:

$$ m_i = |w_i| $$

All eligible parameters across the model are then treated as a single global population.

For a target sparsity (S), a threshold (t) is selected such that approximately the lowest (S) fraction of parameter magnitudes fall below that threshold.

The pruning rule is:

$$ w_i' = \begin{cases} 0, & |w_i| \leq t \ w_i, & |w_i| > t \end{cases} $$

No remaining parameter values are quantized or otherwise numerically transformed.

The model architecture remains unchanged.

Therefore, a sparse checkpoint produced by this experiment still has the same architectural parameter count as the original model. Sparsity refers to the fraction of parameter values that have been set to zero.

---

4. Initial Weight Distribution

Before pruning, the distribution of parameter magnitudes was examined.

Among the parameters eligible for pruning, the minimum observed absolute value was approximately:

$$ 9.895\times10^{-10} $$

and the maximum was:

$$ 9.3125 $$

The fraction of parameters below several magnitude thresholds was:

The result shows that the original model contains very few exact zeros, but a substantial number of relatively small-magnitude parameters.

Small magnitude alone does not imply that a parameter is unnecessary. The pruning experiments below measure the effect of actually removing these parameters.

---

5. Evaluation Setup

Model quality was evaluated using perplexity on the WikiText-2 test split.

For the reported experiments, a fixed subset of:

$$ 50,000 $$

tokens was used.

The same token sequence and evaluation procedure were applied to the original and pruned models.

Perplexity was calculated as:

$$ \mathrm{PPL}

\exp \left( \frac{1}{N} \sum_{i=1}^{N} -\log p(x_i|x_{<i}) \right) $$

Lower perplexity corresponds to better next-token prediction on the evaluation corpus.

The 50,000-token evaluation should be considered a controlled comparative test rather than a complete assessment of the model's capabilities.

---

6. Main Result: 5% Sparsity

The primary checkpoint produced in this experiment targets 5% global sparsity.

The measured values are:

The pruning threshold was:

$$ t = 0.0095214844 $$

The pruning operation replaced:

$$ 6,762,828 $$

parameter values with zero.

The resulting actual sparsity was:

$$ \boxed{5.0276%} $$

The perplexity changed from:

$$ 13.342135 $$

to:

$$ 13.401889 $$

The relative increase in perplexity was approximately:

$$ \boxed{0.448%} $$

No fine-tuning or recovery training was performed after pruning.

---

7. Fine-Grained Sparsity Experiments

To better characterize the low-sparsity regime, additional experiments were performed between 4% and 15% sparsity.

7.1 4–9% sparsity

7.2 9–15% sparsity

The measurements show that degradation is relatively small at low sparsity but becomes increasingly pronounced as more parameters are removed.

---

8. Medium- and High-Sparsity Experiments

Additional measurements extended the pruning range to much higher sparsity levels.

At approximately 62.95% sparsity, the model contains approximately 50 million non-zero parameter values.

However, under the current one-shot magnitude-pruning procedure, the resulting perplexity is extremely high.

This indicates that simply selecting a target of approximately 50M non-zero values is not sufficient to preserve the original model's predictive behavior.

---

9. Interpretation

The experiments indicate that pretrained SmolLM2-135M contains parameter redundancy that can be exploited by simple magnitude pruning.

The most notable result is the low-sparsity regime.

At approximately 5.03% sparsity:

$$ 6.76M $$

parameter values are removed from active computation by setting them to zero, while the measured perplexity increases by only approximately 0.448% on the 50,000-token WikiText-2 evaluation subset.

However, the experiments also show that the cost of additional sparsity is not constant.

The observed progression is approximately:

$$ 5%\rightarrow +0.448% $$

$$ 10%\rightarrow +3.915% $$

$$ 20%\rightarrow +25.357% $$

$$ 30%\rightarrow +118.555% $$

This suggests that magnitude pruning has a limited operating range for this pretrained checkpoint when no recovery training is performed.

---

10. Why Magnitude Pruning Eventually Fails

Global magnitude pruning uses only the scalar quantity:

$$ |w_i| $$

to determine whether a parameter should be removed.

This criterion does not directly model:

• interactions between neighboring parameters;
• layer sensitivity;
• activation magnitudes;
• attention patterns;
• correlated parameter groups;
• redundancy across neurons or channels.

Consequently, a small parameter may still contribute significantly to the computation when considered together with other parameters.

As sparsity increases, the probability of removing collectively important parameter structures also increases.

The experimental results are consistent with this limitation.

---

11. Sparsification vs. Storage Compression

An important distinction is that the current checkpoint demonstrates parameter sparsification, not yet equivalent physical storage compression.

The sparse model still uses a dense safetensors representation.

For example:

[0.82, 0.00, 0.00, 0.91]

is still stored as four parameter values in a dense tensor.

Therefore, the 5.0276% sparsity does not imply a 5.0276% reduction in the checkpoint file size.

Actual storage compression would require a sparse representation that stores non-zero values together with their positions, or another representation specifically designed to exploit zero-valued parameters.

Likewise, sparsity alone does not guarantee an inference speedup. Actual acceleration depends on sparse kernels, hardware support, sparsity structure, and implementation details.

---

12. Reproducibility

All experiments use deterministic model weights and a fixed evaluation procedure.

The primary 5% sparse checkpoint was saved and then independently reloaded from disk.

The saved checkpoint reproduced the previously measured perplexity:

$$ 13.401889 $$

with a numerical difference of approximately:

$$ 1.2\times10^{-7} $$

from the value measured immediately after pruning.

The saved checkpoint contained:

$$ 6,762,828 $$

zero-valued parameter elements, corresponding to:

$$ 5.0276% $$

sparsity.

This provides a direct sanity check that the published checkpoint reproduces the experimental state.

---

13. Limitations

The results should be interpreted within the scope of the experimental setup.

First, the primary evaluation uses 50,000 tokens from WikiText-2 rather than the complete test corpus.

Second, perplexity measures next-token prediction and does not directly measure other capabilities such as instruction following, reasoning, coding, factual recall, or human preference.

Third, the pruning procedure is one-shot global magnitude pruning and does not include recovery training.

Fourth, the experiments do not yet demonstrate reduced checkpoint size or real inference acceleration.

Finally, the results were obtained on one pretrained model checkpoint and should not be assumed to generalize to other architectures or scales without additional experiments.

---

14. Future Work

Several extensions are natural.

Iterative pruning

Instead of removing the target fraction in a single step, pruning can be performed gradually with recovery training between stages:

$$ M_0 \rightarrow M_1 \rightarrow M_2 \rightarrow \cdots $$

This may allow the model to adapt to the increasingly sparse parameter structure.

Layer-wise pruning

Different layers may tolerate different sparsity levels. A layer-sensitive pruning strategy could allocate the pruning budget according to measured importance.

Activation-aware pruning

Instead of using only:

$$ |w_i| $$

parameter importance could be estimated using activation statistics and the contribution of parameters during actual inference.

Structured sparsity

Removing individual weights provides flexible sparsity but can be difficult to accelerate efficiently. Structured pruning may provide a better trade-off between sparsity and hardware efficiency.

Knowledge distillation

A dense teacher could be used to train a sparse student after pruning:

$$ \text{Dense teacher} \rightarrow \text{Sparse student} $$

This could test whether some of the perplexity degradation can be recovered without restoring the removed weights.

Sparse storage and inference

Future work should also measure:

• actual checkpoint size;
• memory usage;
• inference latency;
• throughput;
• hardware utilization.

These measurements are necessary to determine whether the observed mathematical sparsity translates into practical deployment benefits.

---

15. Conclusion

This report evaluated global magnitude pruning on SmolLM2-135M across a broad range of sparsity levels.

The experiments demonstrate that modest sparsity can be introduced into the pretrained model without an immediately proportional degradation in perplexity.

The primary result is:

$$ \boxed{5.0276% \text{ sparsity}} $$

with:

$$ 6,762,828 $$

parameter values set to zero and:

$$ 127,752,180 $$

non-zero parameter values remaining.

On the fixed 50,000-token WikiText-2 evaluation subset:

$$ \mathrm{PPL}_{\mathrm{original}}

13.342135 $$

$$ \mathrm{PPL}_{\mathrm{sparse}}

13.401889 $$

corresponding to an approximately:

$$ \boxed{0.448%} $$

relative increase in perplexity.

The results suggest that simple magnitude pruning can expose measurable parameter redundancy in a pretrained language model, while also showing that increasingly aggressive pruning eventually causes substantial degradation.

The resulting 5% sparse checkpoint therefore serves as a reproducible baseline for further research into more advanced language-model sparsification and compression techniques.

USE🤖

============================================================

Download and Use Sparse SmolLM2-135M

============================================================

!pip -q install -U transformers huggingface_hub safetensors

import torch from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "ALEXFLR/PSmolLM2-135M"

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

------------------------------------------------------------

TOKENIZER

------------------------------------------------------------

tokenizer = AutoTokenizer.from_pretrained( MODEL_ID )

------------------------------------------------------------

MODEL

------------------------------------------------------------

model = AutoModelForCausalLM.from_pretrained( MODEL_ID, dtype=torch.float16 if DEVICE == "cuda" else torch.float32, )

model.to(DEVICE) model.eval()

------------------------------------------------------------

GENERATION

------------------------------------------------------------

prompt = "The future of neural network compression is"

inputs = tokenizer( prompt, return_tensors="pt" ).to(DEVICE)

with torch.no_grad():

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

------------------------------------------------------------

OUTPUT

------------------------------------------------------------

text = tokenizer.decode( outputs[0], skip_special_tokens=True )

print(text)

Download weights in another file