codecourt-solver-grpo-v1

⚖️ CodeCourt Solver — GRPO v1

A LoRA adapter fine-tuned on Qwen2.5-0.5B-Instruct via GRPO (Group Relative Policy Optimization), trained inside the CodeCourt adversarial self-play environment.

The Solver was trained in an adversarial setup where a Setter agent generates coding problems and hidden edge-case traps.

---

Model Description

---

What Is CodeCourt?

Standard coding benchmarks test what a model has memorized. CodeCourt tests what happens when another LLM is actively trying to break it.

The training loop works as follows:

1. Setter (Red Team) generates a coding problem with hidden edge-case traps
2. Solver (Blue Team) — this model — produces a solution
3. Oracle executes the code in a real sandbox and checks all tests including hidden ones
4. Rewards flow: Setter is rewarded when Solver fails; Solver is rewarded when it passes everything

This creates an adversarial training loop where the Solver is optimized against hidden tests rather than a static benchmark.

---

Training Details

Reward Function (5 signals)

solver_reward = (
    correctness_score        # Did ALL tests pass?
  + complexity_match         # Right algorithmic complexity?
  - brute_force_penalty      # O(n²) when O(n log n) expected?
  - hidden_test_regression   # Passed public, failed hidden?
  - unsafe_pattern_penalty   # Suspicious imports caught?
)

---

Results

Training Metrics

Boundary Probe — 6 Adversarial Edge Cases

The committed reference/trained-side artifact reaches 100% on the boundary probe (up from 16.7% baseline). These 6 cases were locked before training began and never exposed during the training loop.

---

How to Use

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-0.5B-Instruct",
    torch_dtype="auto",
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    "ayussssssiiii/codecourt-solver-grpo-v1"
)

# Run inference
prompt = """Solve this coding problem. Write efficient Python code.

Problem: Given an array of integers, find the length of the longest increasing subsequence.

Input: nums = [10, 9, 2, 5, 3, 7, 101, 18]
Expected output: 4
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.1)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

What the Model Learned

After 100 GRPO steps of adversarial self-play training:

• The committed comparison package reports brute-force penalties dropping from 46.7% to 0.0% of episodes
• The committed reference/trained-side artifact reaches 100% on the boundary probe (6/6 adversarial edge cases)
• Best reward reached +34.31 at step 26 (per training_history.json)

---

Limitations

• Small base model (0.5B parameters) — complex multi-step algorithms may still fail
• Training was limited to 100 steps; a longer run with larger max-completion-length is expected to improve stability
• Reward peaked at step 26 then declined — generation length appeared to be a key bottleneck in this run

---

Citation

@misc{codecourt2026,
  author       = {ayussssssiiii},
  title        = {CodeCourt: Adversarial Code Auditing via LLM Self-Play},
  year         = {2026},
  howpublished = {\url{https://github.com/ayushoncode/CodeCourt}},
}

---

Trained inside an adversarial self-play loop — not a static dataset.