Downloads · 30 days
7
78% of all-time downloads
ulab-ai/sotopia-rl-qwen-2.5-7B-grpo
sotopia-rl-qwen-2.5-7B-grpo is a text generation model from ulab-ai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This repository contains the Sotopia-RL model, which is a fine-tuned Large Language Model (LLM) designed for enhanced social intelligence.
Downloads · 30 days
7
78% of all-time downloads
All-time downloads
9
Public
Repo size
41.9 MB
Likes
0
Public
Click a slice to open those files.
.pt20.3 MB · 42%
From the Hugging Face model README
This repository contains the Sotopia-RL model, which is a fine-tuned Large Language Model (LLM) designed for enhanced social intelligence.
Sotopia-RL is a novel framework that refines coarse episode-level feedback from social interactions into utterance-level, multi-dimensional rewards. This approach makes reinforcement learning (RL) more efficient and stable for training socially intelligent agents by addressing challenges like partial observability (where utterances have indirect and delayed effects) and multi-dimensionality (where behaviors like rapport-building or knowledge-seeking contribute indirectly to goal achievement). Experiments show that Sotopia-RL achieves state-of-the-art social goal completion scores in the Sotopia environment.
Sotopia-RL is intended for research and development in training socially intelligent agents, particularly in interactive environments where LLMs need to learn complex social strategies directly through reinforcement learning. It can be used for tasks requiring nuanced social interaction, dialogue generation, and goal-oriented conversations in simulated settings.
This model can be integrated into larger AI systems for applications such as:
This model is not intended for:
The model's performance and behavior are influenced by its training data and the underlying base model. While the multi-dimensional reward design aims to mitigate reward hacking and promote more aligned behaviors, potential risks such as unintended biases, generation of misleading information, or emergent undesirable behaviors still exist. Thorough evaluation and human oversight are recommended for any real-world applications.
Users should be aware of the risks, biases and limitations of the model. It is recommended to:
You can load the base model and the PEFT adapter using the transformers and peft libraries and then merge them for inference.
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel, PeftConfig
import torch
# Load the PEFT configuration
peft_model_id = "Sotopia-RL/Sotopia-RL-Qwen2.5-7B-Instruct" # Replace with the actual repo ID if different
peft_config = PeftConfig.from_pretrained(peft_model_id)
# Load the base model
model = AutoModelForCausalLM.from_pretrained(peft_config.base_model_name_or_path, torch_dtype=torch.bfloat16, device_map="auto")
# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(peft_config.base_model_name_or_path)
# Load the PEFT model
model = PeftModel.from_pretrained(model, peft_model_id)
# Merge LoRA layers with base model and unload LoRA
model = model.merge_and_unload()
# Example inference
messages = [
{"role": "user", "content": "Hello, how are you today? What's your purpose?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
model_inputs.input_ids,
max_new_tokens=50,
do_sample=True,
temperature=0.7,
top_k=50,
top_p=0.95,
)
decoded_output = tokenizer.batch_decode(generated_ids[:, model_inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
print(decoded_output)
The model was trained and evaluated within the Sotopia, an open-ended social learning environment. The framework refines coarse episode-level feedback into utterance-level, multi-dimensional rewards, which form the basis of the training signals. Specific details about proprietary or custom datasets used for fine-tuning are available in the paper.
Sotopia-RL employs a novel framework that refines coarse episode-level feedback into utterance-level, multi-dimensional rewards for reinforcement learning (RL) training. This approach specifically addresses partial observability and multi-dimensionality in social interactions, allowing models to learn sophisticated strategies directly through social interactions.
The model is trained using a novel reward design framework, Sotopia-RL. Specific hyperparameters are detailed in the accompanying paper and codebase.
Evaluations were conducted on existing social intelligence benchmarks within the Sotopia environment.
Experiments were performed in "Sotopia, an open-ended social learning environment," and evaluated on "Sotopia-hard and Sotopia-full" benchmarks.
"Social goal completion scores" were used as a primary metric.
Sotopia-RL achieved significant improvements, demonstrating state-of-the-art social goal completion scores:
These scores significantly outperform existing approaches. Ablation studies confirmed the necessity of both utterance-level credit assignment and multi-dimensional reward design for effective RL training.
If you find our work helpful or inspiring, please feel free to cite it.
@article{ji2025sotopia,
title={Sotopia-RL: Reward Design for Social Intelligence},
author={Ji, Zhoujun and Zhang, Wenqi and Chen, Yutong and Yang, Min and Li, Jiateng and Li, Chen and He, Shuyan and Wei, Yiran and Wang, Tianyi and Xu, Shuyin and Lin, Min and Tian, Tian},
journal={arXiv preprint arXiv:2508.03905},
year={2025}
}
For questions regarding the model, please refer to the project's GitHub repository or contact the authors listed in the paper.