qwen3-gsm8k-lora

Qwen3-0.6B GSM8K LoRA

Model Description

This is a Qwen3-0.6B model fine-tuned on GSM8K math problems using LoRA (Low-Rank Adaptation).

Training Details

• Base Model: Qwen/Qwen3-0.6B
• Dataset: GSM8K (400 samples)
• Method: LoRA fine-tuning
• Epochs: 3
• Batch Size: 1 (with gradient accumulation)

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load base model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "your-username/qwen3-gsm8k-lora")

# Generate response
question = "Tom has 12 apples. He ate 5. How many are left?"
messages = [{"role": "user", "content": question}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=200)
response = tokenizer.decode(outputs[0][len(inputs.input_ids[0]):], skip_special_tokens=True)
print(response)

Training Results

The model shows step-by-step reasoning for basic math problems, though improvements over the base model are limited due to the base model's existing mathematical capabilities.