Downloads · 30 days
0
just1nseo/qwen3-1.7b-if-rlvr-klabl-kl0p005
qwen3-1.7b-if-rlvr-klabl-kl0p005 is a text generation model from just1nseo. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Arm 2 of 4 in a controlled ablation over actor.kllosscoef, the coefficient on the low-variance KL penalty against the frozen reference policy in GRPO. Everything else is held fixed across the four arms; only this coef…
Downloads · 30 days
0
Access
Public
Updated Sep 5, 2026
Repo size
13.8 GB
Likes
0
Public
Click a slice to open those files.
.safetensors13.8 GB · 100%
From the Hugging Face model README
Arm 2 of 4 in a controlled ablation over actor.kl_loss_coef, the coefficient on the
low-variance KL penalty against the frozen reference policy in GRPO. Everything else is held
fixed across the four arms; only this coefficient changes.
Trained with verl on
allenai/IF_multi_constraints_upto5,
reward = fraction of IFEval constraints satisfied. Non-reasoning (enable_thinking=false).
Each global_step_<N>/ folder is a complete bfloat16 model — one per epoch.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "just1nseo/qwen3-1.7b-if-rlvr-klabl-kl0p005"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="global_step_364")
tok = AutoTokenizer.from_pretrained(repo, subfolder="global_step_364")
msgs = [{"role": "user", "content": "Write a product description in exactly 3 sentences."}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
enable_thinking=False) # trained non-reasoning
| step | epoch | held-out IFEval acc |
|---|---|---|
| — | 0 (base) | 0.4652 |
| 91 | 1 | 0.5542 |
| 182 | 2 | 0.6391 |
| 273 | 3 | 0.7111 |
| 364 | 4 | 0.7518 |
512-row held-out split, greedy decoding, n=1.
| metric | value |
|---|---|
| train reward | 0.7552 |
| held-out accuracy | 0.7518 |
| policy entropy | 0.3524 |
| KL from reference | 0.2951 |
| mean response length | 149 tokens |
| gradient norm | 0.2380 |
| arm | kl_loss_coef | held-out acc @364 | KL @364 | resp len | repo |
|---|---|---|---|---|---|
| 2x | 0.002 | 0.7565 | 0.3664 | 133 | kl0p002 |
| 5x | 0.005 | 0.7518 | 0.2951 | 149 | kl0p005 |
| 10x | 0.01 | 0.7283 | 0.1759 | 200 | kl0p01 |
| 20x | 0.02 | 0.6623 | 0.0675 | 322 | kl0p02 |
Findings, briefly:
data.seed=1); rollouts are not.model Qwen/Qwen3-1.7B enable_thinking false
algorithm GRPO kl_loss_type low_var_kl
kl_loss_coef 0.005 use_kl_in_reward false
train_batch_size 1024 rollout n 8
ppo_mini_batch_size 1024 ppo_epochs 1 (implied)
actor lr 5e-7 entropy_coeff 0
max_prompt_length 2048 max_response_length 2048
sampling t=1.0, top_p=1.0, top_k=-1 (untruncated)
train rows 93,881 steps 364 (4 epochs x 91)
hardware 8 x H100 80GB precision fp32 master / bf16 rollout
Because ppo_mini_batch_size == train_batch_size and ppo_epochs == 1, the importance ratio is
identically 1 and PPO clipping never engages (pg_clipfrac == 0.000 at every step). The KL term is
the only guardrail in this recipe — which is what makes the sweep worth running.
git clone https://github.com/ssangyeon/if-rlvr-rlvr
bash if_rlvr/exps/bidirectional/run_qwen3_17b_kl_ablation.sh
Full rationale for every setting: if_rlvr/exps/bidirectional/README_QWEN3_17B_KL_ABLATION.md.