bengali-gpt-poetry

Bengali GPT trained on Free Bengali Poetry

This model is a GPT-style language model trained from scratch on the Free Bengali Poetry dataset. It was implemented in pure PyTorch without using any external NLP libraries.

Model Description

• Architecture: GPT (decoder-only transformer)
• Parameters: 12.67M
• Context length: 256 tokens
• Vocabulary size: 5000 (BPE tokenizer trained on the corpus)
• Training data: 2,686 Bengali poems (public domain)

Intended Use

The model is primarily intended for Bengali poetry generation and as a foundation for further fine‑tuning on downstream Bengali NLP tasks.

Training Details

• Trained for 10 epochs on the dataset (sliding windows of 256 tokens, stride 128).
• Optimizer: AdamW with weight decay, cosine learning rate schedule (warmup 2000 steps).
• Mixed precision (fp16) used on GPU.

Usage

To use the model, you need the custom tokenizer and model definition from the training script. Below is a minimal example:

import torch
import pickle
from your_training_script import GPT, BPETokenizer, Config

# Load tokenizer
with open("bpe_tokenizer.pkl", "rb") as f:
    tokenizer_data = pickle.load(f)
tokenizer = BPETokenizer()
tokenizer.__dict__.update(tokenizer_data)  # restore state

# Load model
config = Config(
    vocab_size=len(tokenizer.vocab),
    block_size=256,
    n_layer=6,
    n_head=6,
    n_embd=384,
    dropout=0.1,
    bias=True
)
model = GPT(config)
model.load_state_dict(torch.load("best_model.pt", map_location="cpu"))
model.eval()

# Generate text
prompt = "আমার সোনার বাংলা"
input_ids = tokenizer.encode(prompt, add_special_tokens=True)
input_tensor = torch.tensor([input_ids], dtype=torch.long)
with torch.no_grad():
    output_ids = model.generate(input_tensor, max_new_tokens=50, temperature=0.8, top_k=40)
print(tokenizer.decode(output_ids[0].tolist()))

Note: You need the original training script (train_bengali_gpt_v1.py)Code to access the GPT, BPETokenizer, and Config classes.

Dataset Citation

If you use this model, please cite the dataset as well:

@misc{ritobrata ghosh_2021, author = {Ritobrata Ghosh}, year = {2021}, title = {Free Bengali Poetry}, publisher = {Kaggle}, address = {Kolkata, India} }

License

The model weights are released under the same license as the dataset: CC BY-SA 4.0.

Contact

For questions or collaborations, feel free to reach out.