Dolphin-Xgen-RL

🐬 Dolphin Xgen RL

Website: https://dphn.ai
Twitter: https://x.com/dphnAI
Web Chat: https://chat.dphn.ai

Overview

In our blog post on training Dphn 405B, we stated that a stronger focus on RL for de-alignment would be crucial. We took a step in this direction by creating a simple LLM-as-judge reinforcement learning environment. During experiments with Xgen 9B instruct, we observed that the model's tone and alignment diverged from what we would normally expect, The model had reward-hacked into improvement. We're releasing this model as a research artifact, as it is not a decensored model.

The model performs best when allowed to reason through problems. Prefilling with <think> brings out its strongest qualities, though it can function without this prompt. Through testing, we found that 16K output tokens with 0.5 temperature and 0.05 min-p yielded the best responses.

Details of the training can be found on our blog: https://blog.dphn.ai/xgen-rl/

Sponsors

Our appreciation for the generous sponsors of Dolphin:

• Lium - provided on-demand 8x B200s for testing and evaluation.

• Andreessen Horowitz - provided a grant that make Dolphin 1.0 possible and enabled us to bootstrap our lab.

What is Dolphin?

Dolphin aims to be a general purpose model, similar to the models behind ChatGPT, Claude, Gemini. But these models present problems for businesses seeking to include AI in their products.

1. They maintain control of the system prompt, deprecating and changing things as they wish, often causing software to break.
2. They maintain control of the model versions, sometimes changing things silently, or deprecating older models that your business relies on.
3. They maintain control of the alignment, and in particular the alignment is one-size-fits all, not tailored to the application.
4. They can see all your queries and they can potentially use that data in ways you wouldn't want. Dolphin, in contrast, is steerable and gives control to the system owner. You set the system prompt. You decide the alignment. You have control of your data. Dolphin does not impose its ethics or guidelines on you. You are the one who decides the guidelines.

Dolphin belongs to YOU, it is your tool, an extension of your will. Just as you are personally responsible for what you do with a knife, gun, fire, car, or the internet, you are the creator and originator of any content you generate with Dolphin.

https://erichartford.com/uncensored-models

Chat Template

We maintained ChatML default chat template for this model. A typical input would look like this:

<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
<|im_start|>assistant

Benchmarks

We used Prime Intellect's environments for evaluation, running 5 passes for MMLU-Pro and 2 passes for all other benchmarks, then averaged the scores. All evaluations were run at 0.5 temperature with 0.05 min-p with 16K output tokens and <think> tag prefilled.

Usage

The model can be used with the following frameworks;

• vllm
• transformers
• llama-cpp