Downloads · 30 days
476
5% of all-time downloads
RichardErkhov/princeton-nlp_-_gemma-2-9b-it-DPO-gguf
princeton-nlp_-_gemma-2-9b-it-DPO-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
476
5% of all-time downloads
All-time downloads
9.6K
Public
Repo size
108 GB
Likes
0
Public
Click a slice to open those files.
.gguf126 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
gemma-2-9b-it-DPO - GGUF
| Name | Quant method | Size |
|---|---|---|
| gemma-2-9b-it-DPO.Q2_K.gguf | Q2_K | 3.54GB |
| gemma-2-9b-it-DPO.IQ3_XS.gguf | IQ3_XS | 3.86GB |
| gemma-2-9b-it-DPO.IQ3_S.gguf | IQ3_S | 4.04GB |
| gemma-2-9b-it-DPO.Q3_K_S.gguf | Q3_K_S | 4.04GB |
| gemma-2-9b-it-DPO.IQ3_M.gguf | IQ3_M | 4.19GB |
| gemma-2-9b-it-DPO.Q3_K.gguf | Q3_K | 4.43GB |
| gemma-2-9b-it-DPO.Q3_K_M.gguf | Q3_K_M | 4.43GB |
| gemma-2-9b-it-DPO.Q3_K_L.gguf | Q3_K_L | 4.78GB |
| gemma-2-9b-it-DPO.IQ4_XS.gguf | IQ4_XS | 4.86GB |
| gemma-2-9b-it-DPO.Q4_0.gguf | Q4_0 | 5.07GB |
| gemma-2-9b-it-DPO.IQ4_NL.gguf | IQ4_NL | 5.1GB |
| gemma-2-9b-it-DPO.Q4_K_S.gguf | Q4_K_S | 5.1GB |
| gemma-2-9b-it-DPO.Q4_K.gguf | Q4_K | 5.37GB |
| gemma-2-9b-it-DPO.Q4_K_M.gguf | Q4_K_M | 5.37GB |
| gemma-2-9b-it-DPO.Q4_1.gguf | Q4_1 | 5.55GB |
| gemma-2-9b-it-DPO.Q5_0.gguf | Q5_0 | 6.04GB |
| gemma-2-9b-it-DPO.Q5_K_S.gguf | Q5_K_S | 6.04GB |
| gemma-2-9b-it-DPO.Q5_K.gguf | Q5_K | 6.19GB |
| gemma-2-9b-it-DPO.Q5_K_M.gguf | Q5_K_M | 6.19GB |
| gemma-2-9b-it-DPO.Q5_1.gguf | Q5_1 | 6.52GB |
| gemma-2-9b-it-DPO.Q6_K.gguf | Q6_K | 7.07GB |
| gemma-2-9b-it-DPO.Q8_0.gguf | Q8_0 | 9.15GB |
base_model: google/gemma-2-9b-it tags:
This model was trained under the same setup as gemma-2-9b-it-SimPO, with the DPO objective.
SimPO (Simple Preference Optimization) is an offline preference optimization algorithm designed to enhance the training of large language models (LLMs) with preference optimization datasets. SimPO aligns the reward function with the generation likelihood, eliminating the need for a reference model and incorporating a target reward margin to boost performance. Please refer to our preprint and github repo for more details.
We fine-tuned google/gemma-2-9b-it on princeton-nlp/gemma2-ultrafeedback-armorm with the DPO objective.
import torch
from transformers import pipeline
model_id = "princeton-nlp/gemma-2-9b-it-DPO"
generator = pipeline(
"text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.bfloat16},
device="cuda",
)
outputs = generator([{"role": "user", "content": "What's the difference between llamas and alpacas?"}], do_sample=False, max_new_tokens=200)
print(outputs[0]['generated_text'])
We use princeton-nlp/gemma2-ultrafeedback-armorm as the preference optimization dataset.
We used the following hyperparameters:
The other hyperparameters are kept the same with our SimPO recipe.
Fine-tuning the google/gemma-2-9b-it on princeton-nlp/gemma2-ultrafeedback-armorm takes around 150 mins to finish on 8xH100 GPUs.
| models | AE2 LC | AE2 WR | AE2 Length | AH | AH Length | GSM | GSM Length | MMLU | MMLU Length |
|---|---|---|---|---|---|---|---|---|---|
| google/gemma-2-9b-it | 51.1 | 38.1 | 1571 | 40.8 | 545 | 87.4 | 395 | 72.7 | 515 |
| princeton-nlp/gemma-2-9b-it-DPO | 67.8 | 65.4 | 2016 | 58.9 | 717 | 88.5 | 392 | 72.2 | 624 |
| princeton-nlp/gemma-2-9b-it-SimPO | 72.4 | 65.9 | 1833 | 59.1 | 693 | 88.0 | 341 | 72.2 | 441 |
The model architecture is based on google/gemma-2-9b-it. We use the DPO training objective.
We used 8xH100 GPUs for model training.
Training was done using the alignment-handbook library.
gemma model:
@article{gemma_2024,
title={Gemma},
url={https://www.kaggle.com/m/3301},
DOI={10.34740/KAGGLE/M/3301},
publisher={Kaggle},
author={Gemma Team},
year={2024}
}
DPO paper:
@article{rafailov2024direct,
title={Direct Preference Optimization: Your language model is secretly a reward model},
author={Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Manning, Christopher D and Ermon, Stefano and Finn, Chelsea},
journal={Advances in Neural Information Processing Systems},
volume={36},
year={2024}
}
SimPO paper:
@article{meng2024simpo,
title={{SimPO}: Simple preference optimization with a reference-free reward},
author={Meng, Yu and Xia, Mengzhou and Chen, Danqi},
journal={arXiv preprint arXiv:2405.14734},
year={2024}
}
UltraFeedback paper:
@article{cui2023ultrafeedback,
title={{UltraFeedback}: Boosting language models with high-quality feedback},
author={Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and Zhu, Wei and Ni, Yuan and Xie, Guotong and Liu, Zhiyuan and Sun, Maosong},
journal={arXiv preprint arXiv:2310.01377},
year={2023}
}
ArmoRM paper:
@article{wang2024interpretable,
title={Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts},
author={Wang, Haoxiang and Xiong, Wei and Xie, Tengyang and Zhao, Han and Zhang, Tong},
journal={arXiv preprint arXiv:2406.12845},
year={2024}
}