RamSundar50M

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This modelcard aims to be a base template for new models. It has been generated using this raw template.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: [Ram Sundar Radhakrishnan]
• Model type: [Language Model based on GPT-2 Architecture]

Uses

Very basic 50M parameter model trained on Bookcorpus and Wikitext as an exercise to understand the basics of LLM design.

Bias, Risks, and Limitations

This model is very small - only 50M parameters. In it's current state, it cannot generate much coherent text.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, GPT2TokenizerFast
m = AutoModelForCausalLM.from_pretrained("i5-8300h/RamSundar50M")
t = GPT2TokenizerFast.from_pretrained("i5-8300h/RamSundar50M")
out = m.generate(**t("Hello Chennai", return_tensors="pt"), max_new_tokens=80, do_sample=True, top_p=0.95, temperature=0.8)
print(t.decode(out[0], skip_special_tokens=True))

Training Details

Training Data

Datasets used: https://huggingface.co/datasets/Salesforce/wikitext https://huggingface.co/datasets/rojagtap/bookcorpus

Training Procedure

Pre-training of the model was done using next-token prediction. Instruction fine-tuning is planned.

Training Hyperparameters

Mixed precision (bf16 on Ampere GPUs; automatic fallback to fp16/fp32 where unsupported). Learning rate was set to 5e-5, with a linear scheduler.

Speeds, Sizes, Times

Training took 9 hours on a rented Vast.ai RTX A2000 instance.

[More Information Needed]

Hardware Used

• Hardware Type: [NVIDIA RTX A2000]
• Hours used: [9]
• Cloud Provider: [Vast.ai]
• Compute Region: [Poland]

Model Card Authors [optional]

Ram Sundar Radhakrishnan

Model Card Contact

[email protected]