Qwen3-4B-OpenThoughts3-Math-SFT-step500

Qwen3-4B OpenThoughts3 Math SFT — Step 500

This is the Hugging Face export of global step 500 from an experimental supervised fine-tuning run of Qwen/Qwen3-4B on the math portion of open-thoughts/OpenThoughts3-1.2M.

The training data contains 103,760 examples built from OpenThoughts3 math traces. Only complete responses with a closed reasoning block and a boxed final answer were retained. The open target format was used: the Qwen3 thinking block is left empty and the source reasoning plus final answer are trained in the visible response channel.

Training configuration

• Checkpoint: global_step_500 (the run was stopped after this checkpoint)
• Objective: full-parameter SFT
• Global batch size: 256
• Micro batch size per GPU: 1 with dynamic batching
• Maximum sequence length: 32,768 tokens
• Optimizer: AdamW
• Learning rate: 5e-6
• Scheduler: cosine, 3% warmup
• Weight decay: 0.01
• Gradient clipping: 1.0
• Precision: bfloat16
• Hardware for this run: 4 GPUs with sequence parallel size 4

Intended use

This checkpoint is intended for research on mathematical reasoning, SFT, and on-policy distillation. It is an intermediate experimental checkpoint, not a production model. AIME evaluation for this checkpoint has not yet been added to this model card.

Inference

Use the tokenizer and chat template shipped in this repository. For the non-thinking behavior used during training, render prompts with enable_thinking=False when supported by your inference stack.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "seomh/Qwen3-4B-OpenThoughts3-Math-SFT-step500"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="auto",
    device_map="auto",
)

Data note

Filtering incomplete OpenThoughts3 generations changes the difficulty distribution: questions whose sampled solutions repeatedly hit the source generation limit are underrepresented. Results should therefore be interpreted as training on the complete-answer subset rather than the full unfiltered math distribution.