qwen2.5-0.5b-codealpaca-lora

Qwen2.5-0.5B CodeAlpaca LoRA

This is a LoRA (Low-Rank Adaptation) fine-tuned adapter for Qwen2.5-0.5B-Instruct, trained on a subset of the CodeAlpaca-20k dataset to improve Python code generation from natural language instructions.

Model Details

• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Fine-tuning method: LoRA (via PEFT)
• Dataset: CodeAlpaca-20k (subset of 50 examples)
• Training hardware: CPU-only (no GPU)
• LoRA config: r=8, alpha=16, target modules: q_proj, v_proj

Intended Use

This adapter is intended for generating short Python functions and code snippets from natural language instructions. It was built as a learning project to understand the fine-tuning pipeline (dataset prep, LoRA, training loop) rather than for production use.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-0.5B-Instruct", dtype=torch.float32
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base_model, "salik702/qwen2.5-0.5b-codealpaca-lora")

prompt = "### Instruction:\nWrite a Python function to check if a number is prime\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• Trained on only 200 examples for a small number of epochs — this is a learning/demo project, not a production-grade coding assistant.
• May overfit to patterns seen in the small training subset rather than generalizing broadly.
• Base model itself (0.5B parameters) has limited reasoning capacity compared to larger code models.

Training Details

Fine-tuned using Hugging Face transformers + peft on a CPU-only machine, using a custom training script with LoRA adapters applied to attention projection layers.

License

MIT