pythia-1B-adapter-wikitext-103

Pythia-1B + AdaLoRA Fine-Tuning on WikiText-103

This repository contains an AdaLoRA fine-tuned version of EleutherAI/pythia-1b trained on the WikiText-103 dataset.
Adaptive LoRA (AdaLoRA) was applied for parameter-efficient fine-tuning on causal language modeling.

---

Model Description

• Base Model: EleutherAI/pythia-1b
• Fine-tuning Method: Adaptive LoRA (AdaLoRA)
• Task: Causal Language Modeling
• Dataset: WikiText-103 (raw v1)

AdaLoRA improves on standard LoRA by dynamically allocating parameter ranks during training, enabling better trade-offs between efficiency and performance.

---

Training Setup

• Framework: Transformers + PEFT + PyTorch
• Adapter Method: AdaLoRA
• Target Modules: query_key_value, dense
• AdaLoRA Config:
  • r=8, init_r=12, target_r=4
  • alpha=64, dropout=0.05
  • orth_reg_weight=0.5
  • Dynamic rank allocation (tinit=100, tfinal=1000, deltaT=10)
• Batch size: 8 (gradient accumulation: 2)
• Sequence length (block size): 1024
• Optimizer: AdamW
• Learning rate: 2e-5 with cosine decay
• Epochs: 10
• Precision: FP16
• Callbacks: Early stopping, custom metric logging

---

Results

Final Training Metrics

• Training Loss: 2.5807
• Final Step Loss: 2.5013
• Gradient Norm: 0.2677
• Learning Rate at End: 1.56e-07

Evaluation Metrics (Epoch 10)

• Evaluation Loss: 2.4908
• Evaluation Perplexity: 12.07
• Evaluation Runtime: 2.1303s
• Samples per Second: 113.13
• Steps per Second: 3.76

---

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
from peft import PeftModel

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("./Pythia-Adapter-wikitext")

# Load base model and AdaLoRA adapters
base_model = AutoModelForCausalLM.from_pretrained("EleutherAI/pythia-1b")
model = PeftModel.from_pretrained(base_model, "./Pythia-Adapter-wikitext")
model.eval()

# Example
input_text = "The history of natural language processing"
inputs = tokenizer(input_text, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(**inputs, max_length=50)
    print("Generated:", tokenizer.decode(outputs[0], skip_special_tokens=True))