lora-t5-pytorch-issues

Model Card for mayankpuvvala/lora-t5-pytorch-issues

This model represents the LoRA adapter weights trained on a custom dataset of PyTorch GitHub issues. It is intended to be used with the base t5-small model to generate issue bodies from titles.

Model Details

Model Description

• Developed by: Mayank Puvvala
• Model type: LoRA Adapter for Text-to-Text Generation
• Language(s): English
• License: MIT
• Fine-tuned from model: t5-small

Model Sources

• Repository: GitHub

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

base_model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
model = PeftModel.from_pretrained(base_model, "mayankpuvvala/lora-t5-pytorch-issues")
tokenizer = AutoTokenizer.from_pretrained("t5-small")

input_text = "Memory leak when using DataLoader with num_workers > 0"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

• Custom dataset of PyTorch GitHub issues, comprising titles and corresponding bodies.

Training Procedure

• Fine-tuned using PEFT with LoRA for 3 epochs.

Training Hyperparameters

• Epochs: 3
• Batch size: 8

Evaluation Metrics

• ROUGE Precision: 53.12%
• ROUGE F1 Score: 49.8%