lmps-challenge-qwen3.5-9b-dpo

Qwen3.5-9B fine-tuned with SFT and DPO for the LM Playschool Challenge

A Qwen/Qwen3.5-9B model fine-tuned via supervised fine-tuning on the success-filtered split of colab-potsdam/playpen-data, followed by an on-policy Direct Preference Optimization pass. DPO preference pairs were constructed on-policy from the SFT model's own successful game rollouts. The model's original response served as the chosen completion, and an artificially corrupted version - padded with verbose filler continuations - served as the rejected one, targeting the common tendency of LLMs to ramble and loop, which can easily downgrade performance in dialogue-based settings. This recipe takes the model from 58.82 (SFT) to 70.21 clemscore. This model was developed as part of a broader investigation into whether DPO can induce novel strategic and rule-following skills in dialogue-game agents, or whether it is better suited to sharpen behaviors already established via SFT. Of the DPO conditions tested, targeting verbosity/looping produced the largest gains, and this checkpoint reflects that setup. See the project's GitHub repo for further information on the training ablations conducted. Furthermore, a technical report will follow, and will be linked to this model card as soon as it is available. Submitted to the LM Playschool Challenge.

Model description

Starting from Qwen/Qwen3.5-9B, the model was supervised-fine-tuned using LoRA on exclusively the successful game rounds present in the interactions subset (train split) of the colab-potsdam/playpen-data. Subsequently, the supervised-fine-tuned model was preference-optimised via DPO on on-policy content-controlled preference pairs, in order to reinforce the model's own successful moves while suppressing verbose and potentially format-breaking prose. The dataset containing the DPO pairs is available here.

Training pipeline

Qwen/Qwen3.5-9B → SFT (LoRA, all linear) → merge adapter into base model → on-policy DPO (LoRA, all linear) → merge adapter into supervised-fine-tuned model. The checkpoint presented in this model card is the final merged model, standalone and directly available for loading.

Training data

• SFT: successful game rounds from the interactions subset of colab-potsdam/playpen-data's train split.
• DPO: self-generated on-policy preference-pairs, constructed starting from the supervised-fine-tuned model's own rollouts on the full clembench benchmark. The SFT model's successful moves (chosen) were paired with the same move plus appended extra prose (rejected). In order to avoid leakage, the clembench game rounds which are part of the playpen eval set were excluded from the pairs creation.

Main results

The table below presents the results obtained by the model described here in the colab-potsdam/playpen-data' evaluation suite, comprising 68 clembench benchmark-2.0 game instances across 14 games, as well as 430 instances of static benchmarks obtained from 5 different benchmarks (bbh, mmlu_pro, cladder, ifeval, eqbench). The clemscore metric quantifies model performance on the clembench game instances, while statscore quantifies that on the static benchmarks. For comparison, results for Qwen/Qwen3.5-9B and the supervised-fine-tuned only version are provided too, along with incremental score differences across training stages (Δ = change relative to the previous row).

Replicating the training

Please refer to the project GitHub repo's README for instructions on how to replicate this model's training.

Hyperparameters

Hyperparameters, training and compute details for the SFT component:

General Training setup

Optimization

LoRA details

Data

Hardware and compute

Training results

Hyperparameters, training and compute details for the DPO component:

General Training setup

Optimization

LoRA details

Data

Hardware and compute

Training results

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "Makaco/lmps-challenge-qwen3.5-9b-dpo"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
  model_name,
  torch_dtype=torch.bfloat16,
  device_map="auto"
)

# The model is specialised for clembench/playpen dialogue games, so it is recommended to use game-oriented prompts
messages = [
  {"role": "user", "content": "You are playing a game of Taboo. The word you have to describe is ocean. "
                              "The taboo words are: sea, water, blue, wave. Reply with 'CLUE: <your clue>'."}
]

inputs = tokenizer.apply_chat_template(
  messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=300, do_sample=False)
response = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)
print(response)

Intended use

Intended for playing text-based dialogue games in the clembench/playpen framework or in similar simulated environments, and for research on interactive dialogue agents and DPO for interaction and game-playing.

Limitations

• Task-specialization: this model was specifically fine-tuned to increase its performance on games in the clembench/playpen framework, therefore could underperform as employed as a general-purpose assistant
• Modest general-capability cost: as highlighted by the main results table, a modest drop in performance on general static benchmarks was recorded in the DPO-fine-tuned model w.r.t the SFT-fine-tuned baseline
• Language: all the game instances on which the model was fine-tuned through its two post-training phases were exclusively in English