Qwen3.5-4B-RL-IF-CC-NGRPO

Qwen3.5-4B-RL-IF-CC-NGRPO

Qwen3.5-4B-RL-IF-CC-NGRPO is an instruction-following reinforcement learning model trained from Qwen/Qwen3.5-4B. Its training data comes from wflying/instruction-following-rl-content-constrained-30k.

In addition to rewards for verifiable format constraints, the model uses an LLM judge to score the semantic correctness and compliance of response content. Training therefore considers both whether a response satisfies the required format and whether its content is valid, helping mitigate reward hacking caused by optimizing format rules alone. The model is further trained with NGRPO (Negative-enhanced Group Relative Policy Optimization), which uses advantage calibration to recover useful training signals from zero-variance negative groups that would otherwise have zero advantage. Asymmetric clipping then balances positive reinforcement and negative penalties, enabling the model to learn more effectively from complex and difficult instructions.

Model details

• Base model: Qwen/Qwen3.5-4B
• Training dataset: wflying/instruction-following-rl-content-constrained-30k
• Training method: LLM-judge-based content-constrained reward + NGRPO
• Primary use: Following complex, content-constrained, and counterintuitive instructions
• Architecture: Qwen3.5, approximately 4.66B parameters, BF16 weights

Method overview

Content-constrained reward

Rewards based only on programmatic rules can encourage a model to satisfy surface-level formatting requirements while ignoring semantic correctness. This model augments format scoring with LLM-judge-based content scoring to evaluate semantic validity and compliance, reducing format-driven reward hacking.

NGRPO

With standard GRPO, a sampled group in which every response receives a low score and the reward variance is zero also receives zero group-relative advantages. These difficult negative examples therefore provide no update signal. NGRPO addresses this issue with:

1. Advantage calibration: Introduces a virtual maximum-reward sample so that zero-variance negative groups receive non-zero advantages and contribute to training.
2. Asymmetric clipping: Expands the upper clipping bound for positive samples and tightens the lower clipping bound for negative samples, reinforcing correct responses while avoiding excessive penalties on negative samples.

Evaluation results

Full evaluation comparison

The table below reports absolute scores for four models across 12 evaluation metrics. IFEval /w content-constrained and IFBench /w content-constrained denote results obtained with semantic content-validity scoring enabled.

Under this evaluation setup, the model achieves a mean score of 62.37, improving by 8.88 points over Qwen3.5-4B (53.49) and by 3.97 points over Qwen3.5-4B-RL-IF-CC (58.40).

Usage example

Use a recent version of transformers with Qwen3.5 support:

import torch
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

model_id = "wflying/Qwen3.5-4B-RL-IF-CC-NGRPO"

model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Explain reinforcement learning in exactly three sentences.",
            }
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    generated_ids = model.generate(**inputs, max_new_tokens=512)

generated_ids = generated_ids[:, inputs.input_ids.shape[1]:]
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])

License

This model inherits the Apache License 2.0 from the base model. Users must also comply with the applicable terms of the training dataset and dependent components.