qwen3-4b-base-rewrite-sft-32k

Qwen3-4B Base rewrite LoRA (32k)

LoRA adapter from SFT on self-rewritten reasoning traces at 32k context, trained on Qwen/Qwen3-4B-Base.

Checkpoint: step 626 (2 epochs).

Load

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-4B-Base"
adapter = "talzoomanzoo/qwen3-4b-base-rewrite-sft-32k"

tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype="auto", device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(model, adapter)

Training

• Base model: Qwen/Qwen3-4B-Base
• Method: LoRA SFT (r=64, alpha=128) via TRL
• Max sequence length: 32768