rl-clarify-orig-prompt-d1-2

rl-clarify-orig-prompt-d1-2

LoRA fine-tune of Qwen2.5-Coder-7B-Instruct trained with PPO-Lagrangian constrained RL on HumanEvalComm.

PPO-Lagrangian RL fine-tune of Qwen2.5-Coder-7B-Instruct (LoRA rank 16) on HumanEvalComm with question budget d1=2.0. Best checkpoint: iter_0029 (val pass@1=0.596, budget feasible; lambda1=0 throughout — budget never binding). Full 417-problem eval: pass@1=0.734, avg_qs=0.741.

Training setup

• Algorithm: PPO with Lagrangian constraint on avg questions per episode
• LoRA rank: 16, alpha 32
• Question budget (d1): 2.0
• Best checkpoint: iter_0029

Checkpoints

Each iter_XXXX/ folder contains LoRA adapter weights and a log.json with per-iteration training metrics (avg_reward, avg_questions, lambda1).

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base, "acv1229/rl-clarify-orig-prompt-d1-2", subfolder="iter_0029")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")