Caracal_GPT

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

Caracal_GPT

This model is a Continuous Pre-trained (CPT) model, adapted from lelapa/InkubaLM-0.4B on the custom dataset. In this new model we add different languages from Kenya, Uganda, South Africa, Western Africa, Somalia, Ethiopia; sub-saharan Africa.

Model description

Caracal GPT is a small causal model that can be used for fine-tuning tasks. It's goal is to be used by the represented language speaker for fine-tuning to a certain language task.

The Pre-trained LM was trained on 20+ African languages, introducing Kenyan lanugages and other languages not widely available in many datasets - Luo (luo), Kamba (kam) , Maasai (mas), Somalia (som), etc.

Intended uses & limitations

The model is to be used for fine-tuning on instruction set data for the given languages.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 32
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 500
• num_epochs: 4
• loss: 1.8

Framework versions

• Transformers 4.45.2
• Pytorch 2.11.0+cu128
• Datasets 2.21.0
• Tokenizers 0.20.3