qwen3-4b-base-distill-sft-32k

Qwen3-4B Base distill LoRA (32k)

LoRA adapter from offline distillation SFT at 32k context on Qwen/Qwen3-4B-Base.

Checkpoint: step 298.

Load

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-4B-Base"
adapter = "talzoomanzoo/qwen3-4b-base-distill-sft-32k"

tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype="auto", device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(model, adapter)

Training

• Base model: Qwen/Qwen3-4B-Base
• Method: LoRA SFT (r=64, alpha=128) via TRL on teacher reasoning traces
• Max sequence length: 32768