Downloads · 30 days
6
11% of all-time downloads
princeton-nlp_-_Llama-3-8B-ProLong-64k-Base-4bits
RichardErkhov/princeton-nlp_-_Llama-3-8B-ProLong-64k-Base-4bits
princeton-nlp_-_Llama-3-8B-ProLong-64k-Base-4bits is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
All-time downloads
53
Public
Parameters
8.1B
6 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors6 GB · 100%
Parameter types
How the weights are stored.
U87B · 86%
Model card
From the Hugging Face model README
Quantization made by Richard Erkhov.
Llama-3-8B-ProLong-64k-Base - bnb 4bits
- Model creator: https://huggingface.co/princeton-nlp/
- Original model: https://huggingface.co/princeton-nlp/Llama-3-8B-ProLong-64k-Base/
Original model description:
license: llama3 datasets:
- princeton-nlp/prolong-data-64K base_model:
- meta-llama/Meta-Llama-3-8B-Instruct
princeton_nlp/Llama-3-8B-ProLong-64k-Base
[Paper] [HF Collection] [Code]
ProLong (<u>Pr</u>incet<u>o</u>n <u>long</u>-context language models) is a family of long-context models that are continued trained and supervised fine-tuned from Llama-3-8B, with a maximum context window of 512K tokens. Our main ProLong model is one of the best-performing long-context models at the 10B scale (evaluated by HELMET).
To train this strong long-context model, we conduct thorough ablations on the long-context pre-training data, SFT data, and numerous other design choices. We demonstrate our findings in our paper, How to Train Long-Context Language Models (Effectively).
Authors: Tianyu Gao*, Alexander Wettig*, Howard Yen, Danqi Chen (* equal contribution)
Contact: {tianyug, awettig}@princeton.edu
The ProLong Models
- princeton_nlp/Llama-3-8B-ProLong-64k-Base ← you are here!
- princeton_nlp/Llama-3-8B-ProLong-64k-Instruct
- princeton_nlp/Llama-3-8B-ProLong-512k-Base
- ⭐ princeton_nlp/Llama-3-8B-ProLong-512k-Instruct
Model card
Here are some quick facts about our main ProLong model: princeton-nlp/Llama-3-8B-ProLong-512k-Instruct.
- Base model: meta-llama/Meta-Llama-3-8B-Instruct
- Long-context continued training: 20B tokens on 64K training data (princeton-nlp/prolong-data-64K), and 20B tokens on 512K training data (princeton-nlp/prolong-data-512K)
- Supervised fine-tuning (SFT): UltraChat
- Maximum context window: 512K tokens
Citation
@article{gao2024prolong,
title={How to Train Long-Context Language Models (Effectively)},
author={Gao, Tianyu and Wettig, Alexander and Yen, Howard and Chen, Danqi},
journal={arXiv preprint arXiv:2410.02660},
year={2024}
}
- Type
- llama
- Created
- Nov 19, 2024
- Updated
- Nov 19, 2024