Qwen3.5-9B-PereGRM-NewReward4D-Step600

Qwen3.5-9B PereGRM NewReward4D — Step 600

This repository contains a full-parameter GRPO checkpoint derived from Qwen/Qwen3.5-9B. The original four-way FSDP checkpoint was merged into standard Hugging Face Transformers safetensors format.

Training snapshot

• Checkpoint: global_step_600
• Run: qwen35_9b_full_grpo_song2_4plus4_len2048_nothink_save300_modelonly_newreward4d_strictbuf_val100_20260827_214901
• Training method: full-parameter GRPO (lora_rank=0)
• Actor GPUs: 4; rollout tensor parallel size: 4
• Train batch size: 4; rollout samples per prompt: 8
• Maximum prompt/response lengths: 512 / 2048 tokens
• Thinking mode: disabled by the chat-template argument used during training
• Actor learning rate: 1e-6
• KL loss: enabled, coefficient 0.001, low_var_kl
• Checkpoint policy: model-only, every 300 steps
• Validation frequency: every 100 steps, using the 100-sample reward validation set

Reward design

The training reward is the equal-weight arithmetic mean of four Judge scores on the original 1–5 scale:

1. Task response quality
2. Memory use
3. Over-personalization restraint
4. Emotional intelligence

The run used a strict bounded reward replay buffer with capacity 1024. Judge requests used the configured gpt-oss-20b endpoints.

Usage

This model requires a recent Transformers version with Qwen3.5 support.

import torch
from transformers import pipeline

pipe = pipeline(
    "image-text-to-text",
    model="KiloHelios/Qwen3.5-9B-PereGRM-NewReward4D-Step600",
    dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {
        "role": "user",
        "content": [{"type": "text", "text": "How can I plan a focused week?"}],
    }
]
result = pipe(text=messages, max_new_tokens=512)
print(result[0]["generated_text"][-1]["content"])

The repository includes the tokenizer, processor configuration, and the chat template saved with the checkpoint.

Limitations

This is a research checkpoint optimized for a personalized-response reward. It has not been established as a general capability or safety improvement. Outputs should be independently evaluated for the intended deployment domain.