Iterative-Qwen-7layers-SGR-Dolci

Iterative-Qwen-7layers-SGR-Dolci

This model is a depth-pruned version of Qwen2.5-7B-Instruct, obtained via Iterative layer pruning and post-trained using QLoRA with Self-Generated Responses (SGR) on the Dolci dataset.

It was released as part of our study on the limits of layer pruning for generative reasoning.

Summary

• Base model: Qwen2.5-7B-Instruct
• Pruning: Iterative
• Depth: 7 layers
• Post-training: QLoRA
• Supervision: Self-Generated Responses (SGR)
• Data: Dolci (SGR variant)

This checkpoint is intended for research and analysis of pruning and recovery, not as a production model.

Paper

On the Limits of Layer Pruning for Generative Reasoning in LLMs
https://arxiv.org/abs/2602.01997

@misc{shrestha2026limitslayerpruninggenerative,
  title={On the Limits of Layer Pruning for Generative Reasoning in LLMs},
  author={Safal Shrestha and Anubhav Shrestha and Aadim Nepal and Minwu Kim and Keith Ross},
  year={2026},
  eprint={2602.01997},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2602.01997},
}