tinyllama-sft-dpo-alpaca-orca

TinyLlama-1.1B — SFT + DPO Fine-Tuned (NLP Assignment)

This is a fully merged fine-tuned version of TinyLlama-1.1B-intermediate-step-1431k-3T trained in two stages as part of an NLP course assignment.

Training Pipeline

Both adapters were merged into the base model weights before upload — no adapter loading needed at inference.

Evaluation Results (10-prompt test set, GPT-4o references)

SFT produced the largest metric gains. DPO improved alignment over the base model but showed some regression on BLEU due to distributional shift between the Orca training format and the Alpaca-style evaluation prompts.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "FarisET/tinyllama-sft-dpo-alpaca-orca",
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("FarisET/tinyllama-sft-dpo-alpaca-orca")

prompt = (
    "Below is an instruction that describes a task. "
    "Write a response that appropriately completes the request.\n\n"
    "### Instruction:\n"
    "Explain the concept of recursion in programming with a simple example.\n\n"
    "### Response:\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
input_len = inputs["input_ids"].shape[1]

with torch.inference_mode():
    out = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(out[0][input_len:], skip_special_tokens=True))

Framework Versions

• transformers 4.44.2
• peft 0.12.0
• trl 0.9.6
• bitsandbytes 0.46.0 (used during training only)
• torch 2.4.x (Google Colab T4 GPU)