Qwen3.5-27B-SFT-SuccessFiltered-PRMGuided

Qwen3.5-27B-sft-ep1

A QLoRA supervised fine-tune of Qwen/Qwen3.5-27B trained on high-scoring game trajectories from the Playpen benchmark (clembench 2.0). This model serves as the policy in a Process Reward Model (PRM) guided inference pipeline, where a companion PRM scores candidate responses at each game turn to select the best action.

Model Details

• Developed by: Diginyx
• Base model: Qwen/Qwen3.5-27B
• Model type: Causal LM — LoRA adapter (PEFT)
• Language: English
• License: Apache 2.0
• Fine-tuning method: QLoRA (4-bit NF4 base + LoRA adapters)
• Training framework: TRL + HuggingFace PEFT

Training Methodology

The model is trained via supervised fine-tuning on game transcripts where the outcome was a win (positive clemscore contribution), filtered from rollouts of the base Qwen3.5-27B-Instruct model playing all clembench 2.0 games. The goal is to teach the policy the turn-level response patterns associated with successful multi-player game trajectories.

Training pipeline:

1. Run the base model on all clembench games to collect rollout transcripts
2. Filter to transcripts with a positive game outcome (win)
3. Fine-tune on the winning turns using QLoRA

Design decisions:

• Positive-only filtering: Using only winning trajectories (rather than all rollouts with reward labels) avoids the model learning from ambiguous partial-credit or losing sequences, keeping the training signal clean.
• 4-bit QLoRA: Reduces VRAM from ~55 GB to ~14 GB, allowing the full 27B model to train on a single 48 GB A40 alongside optimizer states. This makes training accessible without multi-node tensor parallelism.
• LoRA over full fine-tune: Preserves the base model's general language capabilities while adapting the turn-level game response style. The small adapter (r=16) also prevents overfitting on the filtered game corpus.
• Max length 1024: Game turns are typically short; truncating at 1024 tokens keeps the full-vocabulary causal-LM logits tensor (batch × seq × 152k vocab) within GPU memory budget.

Training Data

• Dataset: colab-potsdam/playpen-data — clembench 2.0 game instances (training split)
• Games: All games present in the benchmark (wordle, taboo, reference, clean_up, and others)
• Filtering: Turn-level transcripts where the final game outcome was a win
• Preprocessing: Chat-templated using Qwen3.5 instruction template with enable_thinking=False

Hyperparameters

Compute

Evaluation

Evaluated on the Playpen benchmark (clembench 2.0) using clemscore (quality-weighted success rate across all games) and statscore (static benchmark aggregate).

When used with the companion PRM (Diginyx/Qwen3.5-27B-prm-ep1) in a best-of-N or beam search guided inference setup, this model achieves higher clemscore than the greedy baseline.

Usage

Standalone (greedy inference)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-27B",
    quantization_config=bnb_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "Diginyx/Qwen3.5-27B-sft-ep1")
tokenizer = AutoTokenizer.from_pretrained("Diginyx/Qwen3.5-27B-sft-ep1")

With PRM-guided inference (best-of-N)

Install Playpen and register the model in model_registry.json:

{
  "model_name": "Qwen3.5-27B-sft-ep1",
  "backend": "huggingface_local",
  "huggingface_id": "Qwen/Qwen3.5-27B",
  "model_config": {
    "premade_chat_template": true,
    "load_in_4bit": true,
    "chat_template_kwargs": {"enable_thinking": false},
    "peft_model": "Diginyx/Qwen3.5-27B-sft-ep1"
  }
}

Then run:

python examples/trl/prm_eval.py \
    --policy-model Qwen3.5-27B-sft-ep1 \
    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \
    --game-all \
    --n-candidates 4 \
    --temperature 0.7 \
    --max-tokens 2048

With beam search

python examples/trl/prm_eval.py \
    --policy-model Qwen3.5-27B-sft-ep1 \
    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \
    --mode beam-search \
    --n-candidates 4 \
    --num-beam-iterations 20 \
    --game-all \
    --temperature 0.7 \
    --max-tokens 2048

Companion Models

• PRM: Diginyx/Qwen3.5-27B-prm-ep1 — process reward model trained to score turn-level responses

Framework Versions

• PEFT 0.19.1
• TRL
• Transformers
• bitsandbytes