qwen3-4b-ctf-lite-grpo

Qwen3-4B · ctf-lite curriculum GRPO (LoRA adapter)

A LoRA adapter (rank 32, α 64) for Qwen/Qwen3-4B-Instruct-2507, trained with GRPO (via prime-rl) on a 3-task curriculum from the ctf-lite RL environment — small crypto/forensics capture-the-flag tasks with randomized-per-rollout flags and exact-match rewards.

Result (honest: no significant gain)

Held-out eval, all 9 ctf-lite tasks, same mini_swe_agent harness, n=16 rollouts/task, Fisher's exact:

No result reaches p<0.05 — the difference is within noise. Training reward rose (~0.75→0.82) but did not translate to a significant single-shot held-out gain. An earlier n=3 eval suggested +0.11; that was a small-sample artifact that vanished at n=16. The adapter is published for reproducibility, not as a capability claim. Full writeup + logs + the lever that would actually move it (a procedural difficulty generator): https://github.com/rohseh303/prime-rl-environments (RESULTS.md)

Use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Instruct-2507", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "RSRS64/qwen3-4b-ctf-lite-grpo")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")

Training

GRPO + LoRA, lr 1e-5, group 16, batch 16, ~70 steps, 2×H100. Curriculum = the 3 tasks the base model partially solves (git-leaked-secret, xor-vault, hash-crack) — the only ones with reward variance, hence gradient. Training reward climbed ~0.75 → ~0.82.