smollm2-135m-squad-lora

smollm2-135m-squad-lora

This is a LoRA fine-tuned version of the SmolLM2-135M model trained on 5,000 samples from the SQuAD v1.1 question-answering dataset.
The goal of this fine-tune is to improve extractive question answering on short factual passages.

---

Model Details

• Base model: HuggingFaceTB/smollm2-135m
• Fine-tuning method: LoRA (efficient parameter-efficient fine-tuning)
• Dataset: 5k examples from SQuAD v1.1 (train/dev split)
• Epochs: 2
• Batch size: 4
• Loss after fine-tuning: ~2.21
• Hardware: Google Colab T4 GPU
• Quantization: 4-bit (bitsandbytes)

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "HuggingFaceTB/smollm2-135m"
LORA = "nadaashraff/smollm2-135m-squad-lora"

tokenizer = AutoTokenizer.from_pretrained(BASE)
base_model = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")
model = PeftModel.from_pretrained(base_model, LORA)

prompt = """Read the following context and answer the question.

Context: Egypt ranks as the fifth worst country in the world for religious freedom. According to a 2010 Pew survey, 84% of Egyptians polled supported the death penalty for those leaving Islam.

Question: What percentage of Egyptians polled supported the death penalty for those leaving Islam?
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))