Qwen3-8B-Base-SFT-AM-Thinking-v1-Distilled-Code-600steps

Qwen3-8B-Base-SFT-AM-Thinking-v1-Distilled-Code-600steps

SFT of Qwen/Qwen3-8B-Base on the code subset of AM-Thinking-v1-Distilled (verify_score ≥ 0.9), using the standard Qwen3 chat template and <think>...</think> reasoning protocol.

Training

• Base: Qwen/Qwen3-8B-Base
• Data: AM-Thinking-v1-Distilled (code subset, ~300K samples)
• Hardware: 4 nodes × 8 × H20-96G (32 GPUs)
• Framework: TRL SFTTrainer + FSDP FULL_SHARD + Liger Kernel + FlashAttention-2 + packing
• LR: 5e-5, cosine schedule, warmup_ratio=0.1
• Global batch size: 128 (32 GPUs × bsz=4 × accum=1)
• Max seq len: 32768
• Steps: 600

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "LumosJiang/Qwen3-8B-Base-SFT-AM-Thinking-v1-Distilled-Code-600steps"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "Write a Python function to compute Fibonacci(n)."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=32768,
    do_sample=True,
    temperature=0.6,
    top_p=0.95,
    top_k=20,
)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

The model emits <think>...reasoning...</think> followed by a fenced ```python ``` code block.

Recommended sampling

Aligned with the official Qwen3 sampling protocol: temperature=0.6, top_p=0.95, top_k=20