silapadikaram-gpt

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This modelcard aims to be a base template for new models. It has been generated using this raw template.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: Naveen Raj S (https://www.linkedin.com/in/naveen-raj-s-a7aa7254/)
• Model type: Pre-trained Base Model
• Language(s) (NLP): Tamil
• License: Apache License 2.0

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/nrajmec/silapadikaram

Uses

This is base model which is trained on only Silapadikaram (an Tamil Epic) on character level. This will generate some text in the style of Silapadikaram.

Downstream Use [optional]

Can be further fine-tuned for Tamil literary work

Bias, Risks, and Limitations

Trained only on Silapadikaram, so it may be biased toward the auther's style.

How to Get Started with the Model

Use the code below to get started with the model.

!git clone https://huggingface.co/nrajsubramanian/silapadikaram-gpt
import os
os.chdir('./silapadikaram-gpt')
from transformers import PreTrainedModel, PretrainedConfig

# If you defined your custom classes in a module (like silapadikaram_model.py)
from silapadikaram_model import SilapadikaramModel, SilapadikaramConfig

# Load config & model directly from Hugging Face
repo_id = "nrajsubramanian/silapadikaram-gpt"

config = SilapadikaramConfig.from_pretrained(repo_id)
model = SilapadikaramModel.from_pretrained(repo_id, config=config)
model.eval()  # switch to evaluation mode

import torch

import urllib.request

# download files from Hugging Face repo if needed
!wget https://huggingface.co/nrajsubramanian/silapadikaram-gpt/resolve/main/itos.pt
!wget https://huggingface.co/nrajsubramanian/silapadikaram-gpt/resolve/main/stoi.pt

itos = torch.load("itos.pt")
stoi = torch.load("stoi.pt")

encode = lambda s: [stoi[c] for c in s if c in stoi]
decode = lambda l: ''.join([itos[i] for i in l])

import torch

def generate(model, idx, max_new_tokens=100):
    block_size = 256  # same as training
    for _ in range(max_new_tokens):
        idx_cond = idx[:, -block_size:]
        with torch.no_grad():
            logits = model(idx_cond)['logits'][:, -1, :]
            probs = torch.softmax(logits, dim=-1)
            next_id = torch.multinomial(probs, num_samples=1)
            idx = torch.cat((idx, next_id), dim=1)
    return idx

context = torch.zeros((1, 1), dtype=torch.long)
generated = generate(model, context, max_new_tokens=300)[0].tolist()
print('Model Generated Text:')
print(decode(generated))

Training Details

Training Data

Trained on a static text file of Silapadikaram and it is taken from https://www.projectmadurai.org/pmworks.html

Training Hyperparameters

device = 'cuda' if torch.cuda.is_available() else 'cpu' batch_size = 64 block_size = config.block_size learning_rate = 3e-4 max_iters = 1500 eval_interval = 500 eval_iters = 200

Model Card Authors [optional]

Naveen Raj S ([email protected])