Downloads · 30 days
88
1% of all-time downloads
Sheared-Pythia-160m
princeton-nlp/Sheared-Pythia-160m
Sheared-Pythia-160m is a text generation model from princeton-nlp. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Paper: https://arxiv.org/pdf/2310.06694.pdf Code: https://github.com/princeton-nlp/LLM-Shearing
All-time downloads
16.3K
Public
Repo size
1.4 GB
Likes
4
Public
Repo makeup
Click a slice to open those files.
.bin700 MB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- gpt_neox
- Access
- Public
- Created
- Nov 20, 2023
- Updated
- Dec 4, 2023
- SHA
- c8889f49
Try a prompt
Model card
From the Hugging Face model README
Paper: https://arxiv.org/pdf/2310.06694.pdf
Code: https://github.com/princeton-nlp/LLM-Shearing
License: Must comply with license of Pythia since it's a model derived from Pythia.
Sheared-Pythia-160m is a model pruned and further pre-trained from EleutherAI/pythia-410m. We dynamically load data from different domains in the Pile dataset to prune and contune pre-train the model. We use 0.4B tokens for pruning and 50B tokens for continued pre-training the pruned model. This model can be loaded with HuggingFace via
model = GPTNeoXForCausalLM.from_pretrained("princeton-nlp/Sheared-Pythia-160m")
The model's overall performance is better than EleutherAI/pythia-160m.
Bibtex
@article{xia2023sheared,
title={Sheared llama: Accelerating language model pre-training via structured pruning},
author={Xia, Mengzhou and Gao, Tianyu and Zeng, Zhiyuan and Chen, Danqi},
journal={arXiv preprint arXiv:2310.06694},
year={2023}
}
- Task
- Text Generation
- Library
- transformers
- Type
- gpt_neox
- License
- apache-2.0
- Created
- Nov 20, 2023
- Updated
- Dec 4, 2023