Downloads · 30 days
11
0% of all-time downloads
qwen2.5-7b-base-retool-slime-sft-v2
BillyWang1/qwen2.5-7b-base-retool-slime-sft-v2
qwen2.5-7b-base-retool-slime-sft-v2 is a text generation model from BillyWang1. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
ReTool SFT cold-start of Qwen/Qwen2.5-7B (base, not Instruct), trained with slime on a Megatron backend. This is the tool-format SFT stage that teaches the base model the ReTool interleaved code/tool-call format befor…
All-time downloads
8.2K
Public
Parameters
7.6B
15.2 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors15.2 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- qwen2
- Access
- Public
- Created
- Jun 26, 2026
- Updated
- Jun 26, 2026
- SHA
- 54945161
Try a prompt
Base models
Model card
From the Hugging Face model README
qwen2.5-7b-base-retool-slime-sft-v2
ReTool SFT cold-start of Qwen/Qwen2.5-7B (base, not Instruct), trained with slime on a Megatron backend. This is the tool-format SFT stage that teaches the base model the ReTool interleaved code/tool-call format before RL.
Training configuration
| Setting | Value |
|---|---|
| Base model | Qwen/Qwen2.5-7B (base) |
| Dataset | ReTool-SFT (messages format) |
| Loss | sft_loss, per-token, Qwen loss mask |
| Global batch size | 32 |
| Epochs | 6 (~371 optimizer steps over the ~2k-row corpus, ~62 steps/epoch) |
| Optimizer | Adam (β1=0.9, β2=0.95), weight decay 0.01 |
| LR schedule | 1e-5 → 1e-6, cosine, 10% warmup |
| Precision | bf16, grads all-reduced in fp32 |
| Parallelism | TP=4, PP=1, CP=1, sequence-parallel |
| Hardware | 8× A100-40GB |
| Final train loss | ~0.02 |
The checkpoint corresponds to iteration 371 (end of epoch 6).
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"BillyWang1/qwen2.5-7b-base-retool-slime-sft-v2",
torch_dtype="bfloat16",
device_map="auto",
)
tok = AutoTokenizer.from_pretrained("BillyWang1/qwen2.5-7b-base-retool-slime-sft-v2")
Notes
This is an intermediate SFT cold-start checkpoint intended as the starting point for subsequent RL (GRPO / GFlow-RL) in the ReTool pipeline.
- Task
- Text Generation
- Library
- transformers
- Type
- qwen2
- License
- apache-2.0
- Created
- Jun 26, 2026
- Updated
- Jun 26, 2026