reward-model

Model Overview

• Model Type: Sequence Classification Model for scoring candidate responses
• Architecture: Based on the HuggingFaceTB/SmolLM-135M-Instruct model with an added linear layer for scalar output
• Summary:
  This reward model is trained to assign scores to responses based on human preferences. It learns to distinguish between preferred (chosen) and non-preferred (rejected) responses using the same dataset used for the DPO and PPO training. The output scalar is used as a reward signal during PPO training.

Intended Use

• Use Cases:
  • Scoring and ranking generated responses
  • Serving as a reward function for reinforcement learning methods such as PPO
• Limitations:
  • The model’s performance is closely tied to the quality and diversity of the training data.
  • The single scalar output may not capture all nuances of response quality.

Data

• Training Data:
  HumanLLMs/Human-Like-DPO-Dataset is used. The data is converted into an implicit prompt format that includes both the chosen and rejected dialogues.
• Data Preprocessing:
  The raw data is transformed into a conversation format, where each example consists of a dialogue containing the user prompt and the assistant’s chosen or rejected response.

Training Details

• Optimizer: AdamW
• Learning Rate: 3e-4
• Batch Size: 4
• Training Duration:
  1 epoch (can be extended based on data volume and performance needs)
• Mixed Precision:
  Training uses bf16 mixed precision with the Accelerate framework.

Evaluation and Results

• Metrics:
  During training, the reward model is evaluated by comparing the scores for chosen versus rejected responses, aiming for a significantly higher score for the chosen ones.
• Testing:
  On the test split, the average scores and distribution differences between chosen and rejected responses are analyzed to validate the model’s discrimination capability.

Limitations and Ethical Considerations

• The reward model may reflect biases present in the training data and might not generalize to all dialogue styles.
• Its output should be used as an auxiliary signal for training and not as the sole metric for response quality.

Usage Instructions

• Inference:
  The reward model can be accessed via the HuggingFace Hub and used to score dialogue responses.
• Feedback and Updates:
  Feedback is welcomed to further refine the model and address any biases or shortcomings.