Llama-3-8B-ProLong-512k-Base

princeton-nlp/Llama-3-8B-ProLong-512k-Base

Llama-3-8B-ProLong-512k-Base is a machine learning model from princeton-nlp. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama3.

Downloads · 30 days

13.6K

8% of all-time downloads

All-time downloads

175K

Public

Parameters

32.1 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors32.1 GB · 100%

At a glance

License: llama3
Model type: llama
Access: Public
Created: Aug 22, 2024
Updated: Oct 31, 2024
SHA: 038efd9e

Base models

princeton-nlp/Llama-3-8B-ProLong-64k-Base

Datasets

Model card

From the Hugging Face model README

princeton_nlp/Llama-3-8B-ProLong-512k-Base

[Paper] [HF Collection] [Code]

ProLong (Princeton long-context language models) is a family of long-context models that are continued trained and supervised fine-tuned from Llama-3-8B, with a maximum context window of 512K tokens. Our main ProLong model is one of the best-performing long-context models at the 10B scale (evaluated by HELMET).

To train this strong long-context model, we conduct thorough ablations on the long-context pre-training data, SFT data, and numerous other design choices. We demonstrate our findings in our paper, How to Train Long-Context Language Models (Effectively).

Authors: Tianyu Gao*, Alexander Wettig*, Howard Yen, Danqi Chen (* equal contribution)

Contact: {tianyug, awettig}@princeton.edu

The ProLong Models

Model card

Here are some quick facts about our main ProLong model: princeton-nlp/Llama-3-8B-ProLong-512k-Instruct.

Base model: meta-llama/Meta-Llama-3-8B-Instruct
Long-context continued training: 20B tokens on 64K training data (princeton-nlp/prolong-data-64K), and 20B tokens on 512K training data (princeton-nlp/prolong-data-512K)
Supervised fine-tuning (SFT): UltraChat
Maximum context window: 512K tokens

<img width="80%" alt="image" src="https://github.com/user-attachments/assets/c31c9671-49fe-4776-91d2-de70ffd9f9a1"> ProLong performance on <a href="https://github.com/princeton-nlp/helmet">HELMET</a> averaged over 32K, 64K, and 128K lengths. All models are instruct models. <img width="80%" alt="image" src="https://github.com/user-attachments/assets/a36a7d0f-4480-4a29-80f3-208477707fb7"> ProLong training recipe.

Citation

@article{gao2024prolong,
  title={How to Train Long-Context Language Models (Effectively)},
  author={Gao, Tianyu and Wettig, Alexander and Yen, Howard and Chen, Danqi},
  journal={arXiv preprint arXiv:2410.02660},
  year={2024}
}

Type: llama
License: llama3
Created: Aug 22, 2024
Updated: Oct 31, 2024