llama3_8b_helpful_rm_full

llama3_8b_helpful_rm_full

This model is a reward model (specifically a Helpfulness PRM/RM) introduced in the paper Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards. It is a fine-tuned version of Jennny/llama3_8b_sft_ultrafb designed for simultaneous multi-objective alignment within the MAHALO (Multi-Action-Head ALignment with PRM-guided DecOding) framework.

• Paper: Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards
• Repository: https://github.com/pearls-lab/multiobj-align

Model description

This model serves as a reward signal for the "Helpfulness" dimension in the Human Values domain (open-domain conversations). It follows the MAHALO framework, which standardizes reward training to provide step-level supervision for multi-objective alignment tasks using the UltraFeedback dataset.

Training and evaluation data

The model was trained on the Helpfulness split of the UltraFeedback dataset.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 16
• total_train_batch_size: 512
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 1

Training results

Framework versions

• Transformers 4.45.2
• Pytorch 2.5.1+cu124
• Datasets 3.1.0
• Tokenizers 0.20.3

Citation

@article{shen2025simultaneous,
  title={Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards},
  author={Shen, Yiran and Xia, Yu and Chang, Jonathan and Ammanabrolu, Prithviraj},
  journal={arXiv preprint arXiv:2510.01167},
  year={2025},
  url={https://arxiv.org/abs/2510.01167}
}