pythia-1B-LoRA-wikitext-103

Pythia-1B + LoRA Fine-Tuning on WikiText-103

This repository contains a LoRA fine-tuned version of EleutherAI/pythia-1b trained on the WikiText-103 dataset.
LoRA adapters were applied to efficiently fine-tune the model for causal language modeling while keeping the majority of parameters frozen.

---

Model Description

• Base Model: EleutherAI/pythia-1b
• Fine-tuning Method: Low-Rank Adaptation (LoRA)
• Task: Causal Language Modeling
• Dataset: WikiText-103 (raw v1)

LoRA introduces trainable rank-decomposition matrices into the attention and dense layers of the model, allowing efficient fine-tuning with reduced compute and memory costs compared to full parameter updates.

---

Training Setup

• Framework: Transformers + PEFT + PyTorch
• Adapter Method: LoRA
• Target Modules: query_key_value, dense
• LoRA Config: r=8, alpha=64, dropout=0.1
• Batch size: 8 (gradient accumulation: 2)
• Sequence length (block size): 1024
• Optimizer: AdamW
• Learning rate: 2e-5 with cosine decay
• Epochs: 10
• Precision: FP16
• Callbacks: Early stopping, custom metric logging

---

Results

Final Training Metrics

• Training Loss: 2.4572
• Final Step Loss: 2.4434
• Gradient Norm: 0.4280
• Learning Rate at End: 1.56e-07

Evaluation Metrics (Epoch 10)

• Evaluation Loss: 2.4363
• Evaluation Perplexity: 11.43
• Evaluation Runtime: 2.2418s
• Samples per Second: 107.50
• Steps per Second: 3.57

---

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
from peft import PeftModel

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("./pythia-wikitext-lora")

# Load base model and LoRA adapters
base_model = AutoModelForCausalLM.from_pretrained("EleutherAI/pythia-1b")
model = PeftModel.from_pretrained(base_model, "./pythia-wikitext-lora")
model.eval()

# Example
input_text = "The history of natural language processing"
inputs = tokenizer(input_text, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(**inputs, max_length=50)
    print("Generated:", tokenizer.decode(outputs[0], skip_special_tokens=True))