simple-llm-grpo-lora

Simple LLM — Qwen3.5-4B GRPO

This repository contains a LoRA adapter for Qwen/Qwen3.5-4B. It continues training from thisisandreeeee/simple-llm-sft-lora using Group Relative Policy Optimization (GRPO).

The objective is to produce concise, clear technical answers without sacrificing correctness.

Intended use

Use this experimental adapter for technical explanations, documentation, procedures, and runbooks. Verify generated code, commands, security guidance, and other high-impact content before use.

Training

The base model remained frozen while the existing rank-16 LoRA parameters were trained with GRPO. Training used bf16 on an NVIDIA L4.

Rewards considered correctness, semantic simplicity, clarity, and simplified-technical-English style. Incorrect answers were prevented from outranking correct answers based only on writing style.

Evaluation

Evaluation used 100 held-out English technical prompts.

GRPO produced much shorter and simpler answers with slightly higher technical adequacy, but task fulfilment and clarity declined. This comparison evaluates the complete inference configurations: SFT used adapter scale 0.25, while GRPO used scale 1.0.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3.5-4B"
adapter_id = "thisisandreeeee/simple-llm-grpo-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

Reproducibility

• Code: thisisandreeeee/simple-llm
• Dataset: thisisandreeeee/simple-llm-sft
• Training run: qwen35-4b-grpo-gated-20260912-02
• Evaluation revision: 5038376f35146eb781bc4348cdeb0faca3e582f1