mt5_fine_tuned_model

Fine-Tuned mt5 small Model

Repository Name: sprab4/mt5_fine_tuned_model

Model Description

This repository contains the fine-tuned version of the mT5-Small model, designed for multilingual text-to-text generation tasks. The mT5 architecture supports over 100 languages, including Haitian Creole, making it suitable for text generation, summarization, and instruction-following tasks. Hence, it was fine-tuned for generating text in Haitian Creole by giving prompts in Haitian Creole.

Training Details

• Base Model: google/mt5-small
• Dataset: Haitian Creole dataset from Hugging Face which follows the alpaca format of instructions, input, and output.
• Fine-Tuning Process:
  • Trained for 2 epochs using the Hugging Face Trainer.
  • Validation set was used to evaluate performance periodically during fine-tuning.
  • Instruction-response formats (Alpaca-style) were prioritized in the training dataset.
• Hyperparameters:
  • learning_rate=1e-4
  • per_device_train_batch_size=4
  • gradient_accumulation_steps=4
  • num_train_epochs=2
  • fp16=False
  • save_steps=500
  • save_total_limit=2
  • logging_steps=10
  • optim="adamw_hf"
  • weight_decay=0.05
  • warmup_steps=100

Results

Quantitative Metrics

BERTScore Varies based on the response for each prompt

BERTScore for the below prompt:

Prompt: Solèy la ap kouche ak yon bèl solèy kouche vizib nan orizon an, dekri li nan de fraz

English Translation: The sun is setting and a beautiful sunset is visible on the horizon, describe it in two sentences

• BERTScore Precision: 0.6636
• BERTScore Recall: 0.6542
• BERTScore F1: 0.6589

Example Output

Prompt: Solèy la ap kouche ak yon bèl solèy kouche vizib nan orizon an, dekri li nan de fraz

English Translation: The sun is setting and a beautiful sunset is visible on the horizon, describe it in two sentences

Prediction: ak yon lòd efè yo fè

English Translation: and an order of effect to do