qwen3-8b-rh-sampler-ckpts

102 LoRA checkpoints from a deliberately-induced reward-hacking training run on Qwen3-8B. Misaligned!

The base model was trained with GRPO on a deliberately-weak code-execution environment using a difficulty-filtered dataset (vgel/SYNTHETIC-2-RL-fn_name-Qwen3-32B-Hard), no KL penalty. The model learns to cheat, first by changing tests, then by hacking the harness.

The bulk of reward hacking begins around step 45-55 (AST metrics):

Each step_NNNN/ dir contains:

• adapter_model.safetensors, adapter_config.json — the LoRA adapter (rank 64) for that gradient step
• metrics.json — aggregate training metrics for that step (reward, hack rates, etc.)
• rollout_summaries.jsonl — per-trajectory metrics for all ~368 rollouts at that step
• rollouts_with_chat.jsonl (92 of 102 steps) — full conversation transcripts for a sample of trajectories, useful for inspecting the rationalization-comment fingerprint and seeing the hack motifs in context

Datasets

2 epochs of vgel/SYNTHETIC-2-RL-fn_name-Qwen3-32B-Hard

Code & environment

• GitHub: vgel/simple-reward-hacking
• Prime Intellect environment hub: vgel/simple-reward-hacking

Acknowledgements

This work was supported by the Czech Science Foundation, grant No. 26-23955S.