nanoTharoor

nanoTharoor

A Qwen2.5-3B-Instruct model fine-tuned on the rhetorical style of Shashi Tharoor — Indian politician, author, and former UN Under-Secretary-General, known for his elevated vocabulary, historical analogies, and aphoristic closures.

LoRA implemented from scratch in pure PyTorch — no PEFT during training. The project also includes Fisher Information Matrix analysis to measure whether LoRA implicitly protects the base model from catastrophic forgetting.

Key Finding

Mean Fisher-LoRA Pearson r = 0.002 — essentially zero. The LoRA updates are orthogonal to the parameters the base model relies on for general language capability. LoRA provides implicit forgetting protection by construction for persona fine-tuning (Outcome A).

Training Results

Architecture

• Base: Qwen/Qwen2.5-3B-Instruct
• LoRA rank r=16, alpha=32, dropout=0.05
• Target modules: q_proj, k_proj, v_proj, o_proj (all 36 layers → 144 adapters)
• LR: 2e-4 with cosine warmup

Usage

# Note: adapter.pt uses our custom format — load with lora.py from the repo
# Standard PEFT-compatible release coming soon
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-3B-Instruct",
    torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
# Load adapter: see lora.py in this repo

Analysis plots

See the analysis/ folder for SVD evolution, effective rank heatmap, Fisher-LoRA overlap, and loss curves.