Downloads · 30 days
9
20% of all-time downloads
AlexWortega/qwen35-4b-soyuz-grpo-v3
qwen35-4b-soyuz-grpo-v3 is a reinforcement learning model from AlexWortega. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
GRPO (Group Relative Policy Optimization) fine-tune of Soyuz-4B, where the reward is a separate Claude Code instance grading each answer against an explicit per-task criteria checklist. This card documents how the GRP…
Downloads · 30 days
9
20% of all-time downloads
All-time downloads
44
Public
Parameters
4.2B
8.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors8.4 GB · 100%
From the Hugging Face model README
GRPO (Group Relative Policy Optimization) fine-tune of Soyuz-4B, where the reward is a separate Claude Code instance grading each answer against an explicit per-task criteria checklist. This card documents how the GRPO actually affected the model — including a negative transfer result, reported honestly.
Qwen3_5ForCausalLM) + LoRA r=32, on-policy REINFORCE with a group baseline
(G=6 rollouts/task) + KL to the frozen base (LoRA-disabled reference). Custom bare-torch loop.claude -p (headless Claude Code) scores each rollout passed/total against the
task's explicit criteria → reward in [0,1]. Criteria-checklist grading has much lower variance
than free-form quality scoring (this was the single biggest fix for a stable signal).score + 0.5·(score − score_prev) → rewards climbing out of a bad answer, penalises regression.On the trained distribution — clear gain:
| metric | epoch 1 | epoch 2 | Δ |
|---|---|---|---|
| reward (criteria) | 0.700 | 0.757 | +0.057 (27/44 tasks up) |
| turn-0 score | 0.652 | 0.723 | +0.071 |
| recovery (climb from a bad turn-0) | 0.301 | 0.273 | −0.028 |
The recovery skill internalised into the first turn. Recovery fell not because the model got worse at fixing — but because turn-0 got better, so there were fewer bad starts left to recover from. The model learned to do it right the first time (e.g. product-description turn-0 0.23→0.60, cover-letter 0.3→0.93, ELI5 0.0→0.73). Biggest gains were on the hard tail (semver-compare +0.35, weighted-choice +0.27, streaming-median +0.22, SQL-injection-fix +0.18).
Negative transfer to agentic terminal tasks. Tested on the terminal-bench-2 tasks the base model partially solved, GRPO-v3 regressed: tasks-with-any-pass dropped 11/11 → 4/11 (e.g. hf-model-inference 4/5→0/5, openssl 3/5→0/5, sqlite 3/5→0/5). General single-response coding/creative GRPO does not transfer to — and hurts — multi-step shell-agent behaviour. To improve agentic terminal performance you must run GRPO on the agentic rollouts themselves (verifier reward), not on single-shot coding questions.
App generation. Works well as a single-shot generator (one prompt → full index.html), which
is the model's strength; it fails inside a heavy agent loop (drowns in planning). Suppress its
<think> ("output only the HTML, do not plan") or it spends its token budget reasoning and never
emits the code.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
m = AutoModelForCausalLM.from_pretrained("AlexWortega/qwen35-4b-soyuz-grpo-v3", dtype=torch.bfloat16, device_map="cuda")
tok = AutoTokenizer.from_pretrained("AlexWortega/qwen35-4b-soyuz-grpo-v3")
Best for single-response coding / writing / rewriting. For multi-step shell agents, prefer the base Soyuz-4B.
| Asset | Link |
|---|---|
| Base | qwen35-4b-soyuz-merged |
W&B: https://wandb.ai/alexwortega/fast-qwen35-sft (run soyuz-4b-grpo-recovery-v3)