Downloads · 30 days
7
12% of all-time downloads
LoneStriker/Snorkel-Mistral-PairRM-DPO-3.0bpw-h6-exl2
Snorkel-Mistral-PairRM-DPO-3.0bpw-h6-exl2 is a text generation model from LoneStriker. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
We offer a temporary HF space for everyone to try out the model: - Snorkel-Mistral-PairRM-DPO Space
Downloads · 30 days
7
12% of all-time downloads
All-time downloads
59
Public
Repo size
6 GB
Likes
0
Public
Click a slice to open those files.
.safetensors3 GB · 100%
From the Hugging Face model README
We offer a temporary HF space for everyone to try out the model: -> Snorkel-Mistral-PairRM-DPO Space
We also provide an inference endpoint for everyone to test the model. It may initially take a few minutes to activate, but will eventually operate at the standard speed of HF's 7B model text inference endpoint. The speed of inference depends on HF endpoint performance and is not related to Snorkel offerings. This endpoint is designed for initial trials, not for ongoing production use. Have fun!
import requests
API_URL = "https://t1q6ks6fusyg1qq7.us-east-1.aws.endpoints.huggingface.cloud"
headers = {
"Accept" : "application/json",
"Content-Type": "application/json"
}
def query(payload):
response = requests.post(API_URL, headers=headers, json=payload)
return response.json()
output = query({
"inputs": "[INST] Recommend me some Hollywood movies [/INST]",
"parameters": {}
})
Training dataset: snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset
We utilize ONLY the prompts from UltraFeedback; no external LLM responses used.
This overview provides a high-level summary of our approach. We plan to release more detailed results and findings in the coming weeks on the Snorkel blog.
The prompt format follows the Mistral model:
[INST] {prompt} [/INST]
Unlike our customers, who have very specific use cases to align LLMs to, the AlpacaEval 2.0 leaderboard measures the ability of LLMS to follow user instructions. With this demonstration, we focus on the general approach to alignment. Thus, we use a general-purpose reward model - the performant PairRM model. We use the Mistral-7B-Instruct-v0.2 model as our base LLM.
For interest in building your specialized internal reward models that reflect your enterprises' needs, please contact the Snorkel AI team or consider attending our Enterprise LLM Summit: Building GenAI with Your Data on January 25, 2024 to learn more about "Programmatically scaling human preferences and alignment in GenAI".
On Alpaca-Eval 2.0:
After applying the above methodology:
We recognize that the Alpaca-Eval 2.0 benchmark does not entirely capture the full range of capabilities and performances of LLMs. However, in our current work, where the goal is to align with general "human preferences," Alpaca-Eval 2.0 serves as a suitable and representative benchmark. Moving forward, we anticipate further contributions from the community regarding new alignment axes, and conduct evaluations using other appropriate benchmarks.
The Alpaca-Eval 2.0 evaluator, "gpt-4-turbo," exhibits a bias towards longer responses. This tendency might also be present in our chosen reward model, resulting in our model producing lengthier responses after DPO iterations, which can be among the factors to our higher ranks on the leaderboard. Future work could include measures to control response length and other relevant metrics.
The model is a quick demonstration that the LLMs can be programmatically aligned using smaller specialized reward models. It does not have any moderation mechanisms. We look forward to continuing to engage with the research community and our customers exploring optimal methods for getting models to respect guardrails, allowing for deployment in environments requiring moderated outputs.
Hoang Tran, Chris Glaze, Braden Hancock