Downloads · 30 days
12
25% of all-time downloads
npatel121/qwen2-7b-grpo-gsm8k
qwen2-7b-grpo-gsm8k is a reinforcement learning model from npatel121. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Qwen/Qwen2-7B-Instruct fine-tuned with GRPO (DeepSeek-style, no external RL library) on GSM8K for 1000 iterations, as part of a scaling-laws comparison across 0.5B/1.5B/7B model sizes. See npatel121/qwen2-0.5b-grpo-gs…
Downloads · 30 days
12
25% of all-time downloads
All-time downloads
48
Public
Parameters
7.6B
15.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors15.2 GB · 100%
From the Hugging Face model README
Qwen/Qwen2-7B-Instruct fine-tuned with GRPO (DeepSeek-style, no external RL library) on GSM8K for 1000 iterations, as part of a scaling-laws comparison across 0.5B/1.5B/7B model sizes. See npatel121/qwen2-0.5b-grpo-gsm8k and npatel121/qwen2-1.5b-grpo-gsm8k for the smaller models in the same comparison.
Trained via 2-way data-parallel GRPO (torchrun --nproc_per_node=2 DDP) on a rented 4x H100 NVL instance — each rank held a policy+frozen-reference GPU pair. Getting a 7.6B-parameter model to fit required reducing two hyperparameters from the 0.5B/1.5B baseline, due to real memory constraints at this scale (see the training repo's scaling_analysis.md for the full debugging story):
| Hyperparameter | 0.5B / 1.5B baseline | This 7B run |
|---|---|---|
num_chains (GRPO group size) | 16 | 4 |
max_completion_length | 786 | 400 |
gradient_accumulation_steps | 4 | 8 |
learning_rate | 5e-6 | 5e-6 (unchanged) |
kl_weight_beta | 0.04 | 0.04 (unchanged) |
This is a real deviation from the other two models in the comparison, not just a different model size — treat cross-model comparisons as directional rather than a fully controlled experiment.
num_chains/max_completion_length forced by memory constraints is not resolved by this single run — see scaling_analysis.md in the training repo for the full discussion.Standard GSM8K-style prompting: the model was trained to respond in a strict <reasoning>...</reasoning><answer>INTEGER</answer> XML format (see the training repo's system prompt in rl_datasets.py).
Model weights only (bf16 safetensors) + tokenizer. Optimizer state (training_state.pt, needed only to resume training via --continue_train, not for inference) is not included.