gpt2-medium-LoRA

Model Card for Story Generation Model (Fine-tuned GPT-2 Medium)

Model Details

Model Description

This model is a fine-tuned version of GPT-2 Medium on the BookCorpus dataset using PEFT (LoRA) for parameter-efficient training. It is intended for story generation tasks, producing creative and coherent narrative text given a prompt.

• Developed by: Mahesh Dindur
• Institution: KLE Dr. MSSCET, Belagavi
• Funded by: N/A (individual project)
• Shared by: Mahesh Dindur
• Model type: Causal Language Model (CLM)
• Language(s) (NLP): English
• License: MIT (Open Source)
• Finetuned from model: GPT-2 Medium

Model Sources

• Repository: [N/A]
• Paper: N/A
• Demo: [N/A]

Uses

Direct Use

• Creative story generation based on given prompts.

Downstream Use

• Adaptation to specific genres (e.g., horror, sci-fi, fairy tales).

Out-of-Scope Use

• Real-time decision-making in critical domains.
• Factual knowledge retrieval.

Bias, Risks, and Limitations

• Known limitations:
  • GPT-2 Medium has fewer parameters compared to larger LMs, so logical reasoning ability remains limited even after fine-tuning.
  • Generated text may contain factual inaccuracies or hallucinations.
  • May produce biased or inappropriate outputs due to biases in the training data.

Recommendations

Users should review generated outputs before deployment in any real-world application.

PEFT (Parameter-Efficient Fine-Tuning)

Parameter-Efficient Fine-Tuning is a training approach designed to adapt large pre-trained models to new tasks without updating all their parameters. Instead of fine-tuning the full model, PEFT methods modify only a small set of additional parameters, keeping the rest of the model frozen. This reduces computational requirements, speeds up training, and allows the same base model to be adapted to many tasks efficiently.

LoRA (Low-Rank Adaptation)

Low-Rank Adaptation is a PEFT technique that inserts small, trainable rank-decomposition matrices into the transformer's architecture. Instead of updating the full weight matrices, LoRA learns a low-rank approximation of the weight updates while keeping the original model weights frozen.

Advantages of LoRA include:

• Reduced training cost.
• Lower VRAM usage.
• No need to store a full copy of the fine-tuned model — only adapter weights are saved.
• Easy sharing and merging with other adapters.

How to Get Started with the Model

from transformers import GPT2LMHeadModel, GPT2Tokenizer

model_name = "path_or_repo_id"  # Replace with your repo path or Hugging Face model ID
tokenizer = GPT2Tokenizer.from_pretrained(model_name)
model = GPT2LMHeadModel.from_pretrained(model_name)

prompt = "Once upon a time in a distant galaxy,"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=200, temperature=0.8, top_p=0.95, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

• Dataset: BookCorpus
• Description: A large corpus of books used for language model pretraining, providing diverse narrative text.

Training Procedure

• Preprocessing: Tokenized with GPT-2 tokenizer, applied LoRA adapters via PEFT.
• Training regime: fp16 mixed precision.
• Hardware: 2× NVIDIA RTX A6000 GPUs
• Hours used: [N/A]
• Cloud Provider: On-premise college GPU cluster.
• Compute Region: Belagavi, India.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator.

• Hardware Type: 2× NVIDIA RTX A6000 series GPUs
• Hours used: [N/A]
• Cloud Provider: On-prem (college lab)
• Compute Region: India
• Carbon Emitted: [N/A]

Technical Specifications

• Architecture: Transformer-based GPT-2 Medium with LoRA adapters via PEFT.
• Frameworks: Hugging Face Transformers, PEFT 0.14.0, PyTorch.
• Precision: fp16 mixed precision.

Citation

BibTeX:

[Add citation if applicable]

APA:

[Add citation if applicable]

Model Card Authors

Mahesh Dindur

Model Card Contact

[N/A]