distilgpt2-lora-fact

distilgpt2 LoRA — "Fact:" answer style

A deliberately tiny LoRA adapter, trained as a teaching example. It teaches distilgpt2 one narrow behaviour: answer a Q: prompt with a single-line statement that begins with Fact:.

Trained on 8 hand-written examples in about 7 seconds on an Apple MPS laptop. It is not useful for anything real — it is useful for seeing what the moving parts of LoRA actually do.

What it does

The first answer is memorised verbatim from the training set. The second prompt was never trained on: the Fact: style transferred, but the content is wrong — low-rank matrices do not "set the weights of the edges". That gap is the honest lesson of an 8-example fine-tune. LoRA transferred style, not knowledge.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilgpt2")
base = AutoModelForCausalLM.from_pretrained("distilgpt2")
model = PeftModel.from_pretrained(base, "yennj12/distilgpt2-lora-fact").eval()

inputs = tok("Q: What is LoRA?\nA:", return_tensors="pt")
print(tok.decode(model.generate(**inputs, max_new_tokens=24,
                                do_sample=False,
                                pad_token_id=tok.eos_token_id)[0]))

Call model.merge_and_unload() to fold the adapter into the base weights for zero inference overhead.

Training

The learning rate is ~100x a typical full fine-tuning LR. That is normal for LoRA: you are training freshly initialised matrices, not nudging pretrained weights.

Limitations

Everything about this model is a limitation. It was overfit on purpose to 8 sentences about LoRA, so it will state confident falsehoods on any prompt outside that set, and it inherits all of distilgpt2's biases underneath. Use it to learn how adapters work, not for generation.

Source

Training code and a from-scratch (no-PEFT) reimplementation of the same arithmetic: see the repository this adapter was trained from. Follows the Fine-Tuning with LoRA & QLoRA lesson from AI Engineering from Scratch.