tinyllama-xsum-lora

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

---

license: apache-2.0 base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 library_name: peft tags:

• lora
• peft
• summarization
• text-generation
• tinyllama datasets:
• EdinburghNLP/xsum language:
• en pipeline_tag: summarization

---

TinyLlama-1.1B LoRA — News Summarization (XSum)

This is a LoRA (PEFT) adapter fine-tuned on top of TinyLlama/TinyLlama-1.1B-Chat-v1.0 for single-sentence news summarization, trained on a subset of the XSum dataset.

This is an adapter only — it must be loaded on top of the base model (see Usage below).

Model Details

• Base model: TinyLlama-1.1B-Chat-v1.0
• Fine-tuning method: LoRA (Low-Rank Adaptation) via PEFT
• Task: Abstractive summarization (news articles → one-sentence summary)
• Training data: 3,000 examples sampled from XSum train split
• Trainable parameters: <1% of total model parameters (LoRA rank=16, alpha=32, targeting q/k/v/o attention projections)
• Quantization: 4-bit (NF4) via bitsandbytes during training

Training Procedure

• Epochs: 3
• Batch size: 4 (effective batch size 16 with gradient accumulation)
• Learning rate: 2e-4
• LoRA config: r=16, alpha=32, dropout=0.05, target_modules=["q_proj", "v_proj", "k_proj", "o_proj"]
• Hardware: Single T4 GPU (Google Colab, free tier)

Evaluation

Evaluated on a held-out sample of 30 examples from the XSum validation split, comparing the base model (zero-shot) against the LoRA fine-tuned model.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
adapter_name = "your-username/tinyllama-xsum-lora"  # replace with your repo id

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    quantization_config=bnb_config,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_name)

prompt = (
    "<|system|>\nYou are a helpful assistant that summarizes news articles "
    "in one short sentence.</s>\n"
    "<|user|>\nSummarize the following article:\n{your_article_here}</s>\n"
    "<|assistant|>\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=60, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

• Trained on a small subset (3,000 examples) of XSum for a short training run — not intended to match dedicated summarization models (e.g. BART, PEGASUS) in quality.
• Base model (TinyLlama-1.1B) is small; summaries may occasionally be generic or miss key details from longer articles.
• Evaluated on a small sample (30 examples); ROUGE scores are indicative, not a rigorous benchmark.

Intended Use

Educational / portfolio project demonstrating parameter-efficient fine-tuning (LoRA/PEFT) of a small LLM for a summarization task under limited compute (free-tier Colab GPU).