Downloads · 30 days
419
5% of all-time downloads
allenai/llama-3-tulu-2-8b-uf-mean-rm
llama-3-tulu-2-8b-uf-mean-rm is a text classification model from allenai. Use it when you need a label for a piece of text. It is set up for transformers. The card lists the license as apache-2.0.
<center <img src="https://huggingface.co/datasets/allenai/blog-images/resolve/main/tulu-2.5/tulu25banner.png" alt="Tulu 2.5 banner image" width="800px"/ </center
Downloads · 30 days
419
5% of all-time downloads
All-time downloads
8.9K
Public
Parameters
7.5B
15 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors15 GB · 100%
From the Hugging Face model README
Tulu is a series of language models that are trained to act as helpful assistants. This is a 8B reward model used for PPO training trained on the UltraFeedback dataset.
For more details, read the paper: Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback.
Built with Meta Llama 3!
Note that Llama 3 is released under the Meta Llama 3 community license, included here under llama_3_license.txt.
We evaluate the model on RewardBench:
| Model | Score | Chat | Chat Hard | Safety | Reasoning |
|---|---|---|---|---|---|
| Llama 3 Tulu 2 8b UF RM (this model) | 73.6 | 95.3 | 59.2 | 57.9 | 82.1 |
| Llama 3 Tulu 2 70b UF RM | 71.0 | 86.3 | 56.1 | 58.9 | 82.7 |
ultrafeedback_mean_aspects split.The model is trained to use the following format (note the newlines):
<|user|>
Your message here!
<|assistant|>
For best results, format all inputs in this manner. Make sure to include a newline after <|assistant|>, this can affect generation quality quite a bit.
We have included a chat template in the tokenizer implementing this template.
The model was initially fine-tuned on a filtered and preprocessed of the Tulu V2 mix dataset, which contains a diverse range of human created instructions and synthetic dialogues generated primarily by other LLMs. We then further trained the model with a Jax RM trainer built on EasyLM on the dataset mentioned above. This model is meant as a research artefact.
The following hyperparameters were used during PPO training:
If you find Tulu 2.5 is useful in your work, please cite it with:
@misc{ivison2024unpacking,
title={{Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback}},
author={{Hamish Ivison and Yizhong Wang and Jiacheng Liu and Ellen Wu and Valentina Pyatkin and Nathan Lambert and Yejin Choi and Noah A. Smith and Hannaneh Hajishirzi}}
year={2024},
eprint={2406.09279},
archivePrefix={arXiv},
primaryClass={cs.CL}
}