distilgpt2-finetuned-demo

DistilGPT2 Fine-Tuned Demo (Custom Text)

This repository contains a fine-tuned DistilGPT2 language model, trained on a small domain-style corpus with educational / AI agent related text.

Model

• Base model: distilgpt2
• Task: Causal language modeling (next token prediction)
• Library: transformers
• Framework: PyTorch

Training Details

• Epochs: 1

• Batch size: 2 (with gradient accumulation)

• Learning rate: 5e-5

• Weight decay: 0.01

• Dataset: Small custom text corpus (educational / AI topics)

• Eval loss: 5.8886

• Approx. perplexity: 360.89

Usage

You can load and generate text with this model using transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "REPLACE_WITH_YOUR_USERNAME/REPLACE_WITH_REPO_NAME"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "In the future, AI personal assistants will"
inputs = tokenizer(prompt, return_tensors="pt")

output_ids = model.generate(
    **inputs,
    max_length=80,
    do_sample=True,
    top_p=0.9,
    temperature=0.8,
)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))