SmolLM-135M-SFT-DPO

SmolLM-135M — SFT + DPO Fine-Tuned

A fine-tuned version of SmolLM-135M for instruction following, trained using Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO) with LoRA adapters.

Model Details

• Base model: HuggingFaceTB/SmolLM-135M (135M parameters)
• Fine-tuning method: SFT (LoRA) → DPO (LoRA)
• SFT dataset: databricks/databricks-dolly-15k (6,000 samples)
• DPO dataset: Intel/orca_dpo_pairs (3,000 samples)
• Developed by: Areeba Fatima (IBA Karachi — NLP with Deep Learning, Assignment 04)
• Language: English
• License: Apache 2.0

Training Details

Best SFT Configuration (Trial 3)

• LoRA rank: 32, alpha: 64
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Learning rate: 2e-4, epochs: 2, batch size: 2
• Validation loss: 2.2008

Best DPO Configuration

• Beta: [your best beta]
• Learning rate: [your best LR]
• Epochs: [your best epochs]
• Validation loss: [your val loss]

Evaluation Results (10-prompt test set)

How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM-135M")
tokenizer = AutoTokenizer.from_pretrained("AREEBAFATIMA12/SmolLM-135M-SFT-DPO")
model = PeftModel.from_pretrained(base, "AREEBAFATIMA12/SmolLM-135M-SFT-DPO")
model.eval()

prompt = "<|user|>\nWhat causes seasons on Earth?</s>\n<|assistant|>\n"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=150, do_sample=False,
                         pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))