Downloads · 30 days
269
3% of all-time downloads
RichardErkhov/nota-ai_-_cpt_st-vicuna-v1.3-1.5b-ppl-gguf
nota-ai_-_cpt_st-vicuna-v1.3-1.5b-ppl-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
269
3% of all-time downloads
All-time downloads
9.5K
Public
Repo size
17.2 GB
Likes
0
Public
Click a slice to open those files.
.gguf19.9 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
cpt_st-vicuna-v1.3-1.5b-ppl - GGUF
Original model description:
Shortened LLM is a depth-pruned version of large language models for efficient text generation.
| Source<br>Model | Pruning<br>Ratio | Pruning<br>Criterion | HF Models<br>Link |
|---|---|---|---|
| Vicuna-v1.3-7B | 20% | PPL | nota-ai/cpt_st-vicuna-v1.3-5.5b-ppl |
| Vicuna-v1.3-7B | 45% | PPL | nota-ai/cpt_st-vicuna-v1.3-3.7b-ppl |
| Vicuna-v1.3-7B | 60% | PPL | nota-ai/cpt_st-vicuna-v1.3-2.7b-ppl |
| Vicuna-v1.3-7B | 80% | PPL | nota-ai/cpt_st-vicuna-v1.3-1.5b-ppl |
Zero-shot performance over the course of training for models from Vicuna-7B-v1.3 at different pruning ratios. For each model size, the CPT duration was limited to a two-week period, but additional training could further improve the quality.
<img alt="results" img src="https://netspresso-research-code-release.s3.us-east-2.amazonaws.com/compressed-llm/st_llm-cpt_learning-curve.png" width="100%"> </details>| Source<br>Model | Pruning<br>Ratio | Pruning<br>Criterion | HF Models<br>Link |
|---|---|---|---|
| LLaMA-1-7B | 20% | PPL | nota-ai/st-llama-1-5.5b-ppl |
| LLaMA-1-7B | 20% | Taylor+ | nota-ai/st-llama-1-5.5b-taylor |
| Vicuna-v1.3-7B | 20% | PPL | nota-ai/st-vicuna-v1.3-5.5b-ppl |
| Vicuna-v1.3-7B | 20% | Taylor+ | nota-ai/st-vicuna-v1.3-5.5b-taylor |
| Vicuna-v1.3-13B | 21% | PPL | nota-ai/st-vicuna-v1.3-10.5b-ppl |
| Vicuna-v1.3-13B | 21% | Taylor+ | nota-ai/st-vicuna-v1.3-10.5b-taylor |
@article{kim2024shortened,
title={Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods},
author={Kim, Bo-Kyeong and Kim, Geonmin and Kim, Tae-Ho and Castells, Thibault and Choi, Shinkook and Shin, Junho and Song, Hyoung-Kyu},
journal={arXiv preprint arXiv:2402.02834},
year={2024},
url={https://arxiv.org/abs/2402.02834}
}
@article{kim2024mefomo,
title={Shortened LLaMA: A Simple Depth Pruning for Large Language Models},
author={Kim, Bo-Kyeong and Kim, Geonmin and Kim, Tae-Ho and Castells, Thibault and Choi, Shinkook and Shin, Junho and Song, Hyoung-Kyu},
journal={ICLR Workshop on Mathematical and Empirical Understanding of Foundation Models (ME-FoMo)},
year={2024},
url={https://openreview.net/forum?id=18VGxuOdpu}
}