Downloads · 30 days
0
Revot/qwen3.5-4b-grpo-itall144-no-opp
qwen3.5-4b-grpo-itall144-no-opp is a reinforcement learning model from Revot. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
GRPO RL fine-tune of Qwen/Qwen3.5-4B on the ItAll144 iterated 2×2 game-theory benchmark, with opponent descriptions stripped from prompts (ablation row of the SanctGym matrix). LoRA adapters; checkpoints are saved wit…
Downloads · 30 days
0
Access
Public
Updated May 17, 2026
Repo size
10.8 GB
Likes
0
Public
Click a slice to open those files.
.md5.6 KB · 79%
From the Hugging Face model README
GRPO RL fine-tune of Qwen/Qwen3.5-4B on the ItAll144 iterated 2×2 game-theory benchmark, with opponent descriptions stripped from prompts (ablation row of the SanctGym matrix). LoRA adapters; checkpoints are saved with the LoRA merged into the base weights for direct loading.
Trained by Arth Singh, in collaboration with Pepijn Cobben and Colomban Duclaux.
| Branch | Step | Status |
|---|---|---|
step-25 | 25 | First usable checkpoint |
step-50 | 50 | Recommended — plateau checkpoint |
step-75 | 75 | Coming soon (training in progress) |
| Base model | Qwen/Qwen3.5-4B (hybrid Mamba + Transformer, 4.66B params) |
| Algorithm | GRPO (multi-turn agentic rollout via verl + vLLM) |
| LoRA | rank = 32, α = 64, targeting attention projections |
| Optimizer | AdamW, lr = 1e-5 |
| KL penalty | low_var_kl, coef = 0.001 |
| Batch | train_batch_size = 32, rollout n = 8 (256 episodes/step) |
| Hardware | 4 × NVIDIA B200 (192 GB HBM3e), FSDP2, vLLM TP = 2 |
| Per-step time | ~22 min |
| Total epochs scheduled | 1 (~787 steps) — training stopped at step 75 (past convergence) |
tit_for_tat, grim_trigger, always_coop, always_defect, pavlov, fictitious_play, level1)Data prep script:
python examples/data_preprocess/itall144_dataset.py --no_opponent_description --local_save_dir ~/data/itall144_no_opp
| Step | critic/score/mean | actor/entropy | response_length/mean |
|---|---|---|---|
| 1 | 19.79 | 0.249 | 11,184 |
| 25 | 21.03 | 0.168 | 9,145 |
| 35 | 26.52 | 0.140 | 6,928 |
| 50 | 26.56 | 0.092 | 5,606 |
| 55 | 26.13 | 0.078 | 6,099 |
Recommended checkpoint: step-50 (captures the plateau without entropy collapse).
At step 60, 174 / 256 rollouts (68%) emit an empty <think></think> block — the model skips the dedicated reasoning region. The reasoning didn't disappear, it relocated: chain-of-thought style deliberation now appears outside the <think> tags in the regular assistant response. Post-</think> content for skipped-thinking rollouts averages 17K chars (vs 10K when <think> is non-empty).
So the trained policy still reasons; it just stopped using the <think> delimiter.
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"Revot/qwen3.5-4b-grpo-itall144-no-opp",
revision="step-50",
dtype="auto",
device_map="auto",
)
tok = AutoTokenizer.from_pretrained("Revot/qwen3.5-4b-grpo-itall144-no-opp", revision="step-50")
from transformers import AutoModelForCausalLM
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", dtype="auto")
model = PeftModel.from_pretrained(
base,
"Revot/qwen3.5-4b-grpo-itall144-no-opp",
revision="step-50",
subfolder="lora_adapter",
)
vllm serve Revot/qwen3.5-4b-grpo-itall144-no-opp \
--revision step-50 \
--dtype bfloat16 \
--tensor-parallel-size 2
Evaluation is run via the ItAll144 baseline script at N_PER_COMBO=1 (1008 episodes = 7 opponents × 144 games × 1 episode/combo), with opponent descriptions removed at eval time too (matching training):
NO_OPPONENT_DESCRIPTION=1 \
N_PER_COMBO=1 \
MODEL_NAME=Revot/qwen3.5-4b-grpo-itall144-no-opp \
REVISION=step-50 \
bash evals/itall144/run_baseline_eval.sh
Each branch root contains:
model.safetensors — merged base + LoRA weights (~10.4 GB, bf16)lora_adapter/adapter_config.json + adapter_model.safetensors — LoRA-only (~156 MB)config.json, tokenizer.json, chat_template.jinja, etc. — standard HF artifactsuse_remove_padding=False to match Colomban's tested baseline — slightly less memory-efficient at inference but consistent with the SanctGym Col-2 prior runs.fla, causal-conv1d) were not used during training — fell back to pure-torch implementation for hybrid Mamba layers. Speed impact ~10-20% but no accuracy difference.Built on verl (Sheng et al., 2025) and the SanctGym game-theory RL framework.