rl-clarify-orig-prompt-d1-0-again

rl-clarify-orig-prompt-d1-0-again

LoRA fine-tune of Qwen2.5-Coder-7B-Instruct trained with PPO-Lagrangian constrained RL on HumanEvalComm.

PPO-Lagrangian RL fine-tune of Qwen2.5-Coder-7B-Instruct (LoRA rank 16) on HumanEvalComm with question budget d1=0. Second independent training run ('again'). Best checkpoint: iter_0059 (fallback — budget not fully satisfied; avg_qs=0.135 at val). Full 417-problem eval: pass@1=0.601, avg_qs=0.201.

Training setup

• Algorithm: PPO with Lagrangian constraint on avg questions per episode
• LoRA rank: 16, alpha 32
• Question budget (d1): 0.0
• Best checkpoint: iter_0059

Checkpoints

Each iter_XXXX/ folder contains LoRA adapter weights and a log.json with per-iteration training metrics (avg_reward, avg_questions, lambda1).

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base, "acv1229/rl-clarify-orig-prompt-d1-0-again", subfolder="iter_0059")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")