Downloads · 30 days
21
14% of all-time downloads
s-m-sharjeel/qwen2.5-0.5b-alpaca-sft-lora
qwen2.5-0.5b-alpaca-sft-lora is a text generation model from s-m-sharjeel. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
A LoRA adapter that fine-tunes Qwen/Qwen2.5-0.5B for instruction following, using a 5,000-sample subset of the yahma/alpaca-cleaned dataset.
Downloads · 30 days
21
14% of all-time downloads
All-time downloads
145
Public
Repo size
17.3 MB
Likes
0
Public
Click a slice to open those files.
.json11.4 MB · 66%
From the Hugging Face model README
A LoRA adapter that fine-tunes Qwen/Qwen2.5-0.5B for instruction following, using a 5,000-sample subset of the yahma/alpaca-cleaned dataset.
This is the best-performing Alpaca configuration (Trial 3) from a 5-trial LoRA hyperparameter search conducted for an NLP with Deep Learning course assignment (Institute of Business Administration, Karachi).
This is a LoRA adapter, not a standalone model. Load the base model first, then apply the adapter:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
base_model_id = "Qwen/Qwen2.5-0.5B"
adapter_id = "s-m-sharjeel/qwen2.5-0.5b-alpaca-sft-lora"
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)
model.eval()
prompt = "### Instruction:\nWrite a short definition of machine learning.\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=150, do_sample=False,
pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
The model was trained on the Alpaca-style template. For best results, format prompts as:
### Instruction:
{your instruction here}
### Response:
If the task includes additional input/context, use:
### Instruction:
{your instruction here}
### Input:
{context here}
### Response:
| Setting | Value |
|---|---|
| Base model | Qwen/Qwen2.5-0.5B |
| Method | Supervised Fine-Tuning (SFT) with LoRA |
| Dataset | yahma/alpaca-cleaned (5,000-sample subset, seed=42) |
| Train / Validation split | 4,500 / 500 (90/10) |
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| Target modules | q_proj, v_proj, k_proj |
| LoRA dropout | 0.05 |
| Learning rate | 1e-4 |
| LR scheduler | cosine (warmup ratio 0.05) |
| Epochs | 1 |
| Batch size | 4 (gradient accumulation 4 → effective 16) |
| Max sequence length | 512 |
| Precision | fp16 |
| Platform | Kaggle (NVIDIA T4 GPU) |
Evaluated on a held-out set of 10 manually written instruction prompts with reference answers, using BLEU (sacreBLEU) and BERTScore F1. This configuration was selected as the best of 5 Alpaca trials by combined BLEU + BERTScore (validation loss as tie-breaker).
| Model | Mean BLEU | Mean BERTScore F1 |
|---|---|---|
| Base Qwen2.5-0.5B | 6.57 | 0.8854 |
| This model (Alpaca SFT, Trial 3) | 7.27 | 0.8864 |
This represents a +10.7% relative improvement in BLEU over the base model.
Developed as a course assignment for NLP with Deep Learning, Institute of Business Administration (IBA), Karachi.
| Name | ERP |
|---|---|
| Shazain | 27115 |
| Shayan | 26289 |
| Sharjeel | 26932 |
Released under the Apache 2.0 license, matching the base model Qwen2.5-0.5B.