goldberg-grpo-from-sft-v1

goldberg-grpo-from-sft-v1

A LoRA adapter for Qwen/Qwen2.5-Coder-1.5B-Instruct, trained to write Python code that builds 2D Rube Goldberg machines (simulated with pymunk) using a small 6-primitive DSL (ball, box, pin_joint, pivot_joint, motor, spring). Part of the goldberg project.

Training recipe

Two stages, both with LoRA (r=16, alpha=32) on q/k/v/o_proj + gate/up/down_proj:

1. SFT warmup -- 400 synthetic (prompt, working-code) examples generated by a deterministic geometry solver (no LLM involved), verified against the real reward pipeline (87% real success rate before training). 3 epochs.
2. GRPO -- continued from the SFT adapter, 200 steps, group size 8, against the project's own reward function (score_machine()): a real pymunk physics simulation checks whether exactly one ball ends at rest in the goal region, with an unbounded complexity bonus for genuine multi-mechanism chains and penalties against reward-hacking exploits (goal-camping, ball-flooding, energy-gain artifacts, tunneling).

Results (stage-1 eval set, n=20, seed=0)

Reward is not literally accuracy -- see the project's reward.py for the full gate * (1 + complexity) * penalty_multiplier formula -- but success_rate is the direct, literal fraction of eval specs where the generated code actually got a ball to rest in the goal region when simulated.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "briscoooe/goldberg-grpo-from-sft-v1")

# Build the prompt with goldberg.prompting.build_prompt(spec) -- see the repo --
# then generate with model.generate(...) as usual.

Requires the goldberg project's own prompt format and DSL/sandbox/reward pipeline to be meaningful -- see github.com/briscoooe/goldberg.

Framework versions

• PEFT 0.20.0
• TRL: 0.29.1
• Transformers: 4.56.2
• Pytorch: 2.6.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Citations

Cite GRPO as:

@article{shao2024deepseekmath,
    title        = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
    author       = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
    year         = 2024,
    eprint       = {arXiv:2402.03300},
}

Cite TRL as:

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}