Qwen3-4B-RDAN-GRPO

Qwen3-4B-RDAN-GRPO

Checkpoints from RDAN-GRPO (Rubric Decoupled Advantage Normalization GRPO), a rubric-based RL method for instruction following, trained on Qwen3-4B-Instruct-2507.

Each folder is a complete, independently loadable checkpoint.

What it is for

Instructions carry two kinds of requirement: objectively checkable ones ("start with 'My answer'", "under 500 words") and matters of judgment (is it well written). Reward judged quality directly and the policy learns that good prose earns credit while the hard constraints become optional. RDAN-GRPO prevents that.

Method

Sample G = 8 responses per prompt, score each twice, and normalize the two scores over different populations:

A_out  = (r_out  - mu)   / sigma      over all G responses
A_proc = (r_proc - mu_C) / sigma_C    over constraint-satisfying responses only
A      = A_out + A_proc

The second denominator is the whole idea. A response failing any hard constraint is excluded from the process statistics entirely, so quality is a tiebreaker among compliant responses and never a substitute for compliance. It also keeps learning alive once the outcome channel saturates and carries no gradient.

Results

Full official datasets, greedy decoding, each benchmark scored by its own evaluator.

Gain over base at step 200: +4.07 IFEval, +4.76 IFBench, +15.42 MulDimIF.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "beingamanforever/Qwen3-4B-RDAN-GRPO"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="step-000200", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="step-000200")

messages = [{"role": "user", "content": "Write exactly two friendly sentences about rain."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=512)[0], skip_special_tokens=True))

Trained with thinking disabled; evaluate the same way.

Training