Transformers

README

Overview

This project implements a language translation model using GPT-2, capable of translating between Icelandic and English. The pipeline includes data preprocessing, model training, evaluation, and an interactive user interface for translations.

Features

Text Preprocessing: Tokenization and padding for uniform input size.

Model Training: Training a GPT-2 model on paired Icelandic-English sentences.

Evaluation: Perplexity-based validation of model performance.

Interactive Interface: An easy-to-use widget for real-time translations.

Installation

Prerequisites

Ensure you have the following installed:

Python (>= 3.8)

PyTorch

Transformers library by Hugging Face

ipywidgets (for the translation interface)

Steps

Clone the repository:

git clone <repository_url> cd <repository_name>

Install the required libraries:

pip install -r requirements.txt

Ensure GPU availability for faster training (optional but recommended).

Usage

Training the Model

Prepare your dataset with English-Icelandic sentence pairs.

Run the script to preprocess the data and train the model:

python train_model.py

The trained model and tokenizer will be saved in the ./trained_gpt2 directory.

Evaluating the Model

Evaluate the trained model using validation data:

python evaluate_model.py

The script computes perplexity to measure model performance.

Running the Interactive Interface

Launch a Jupyter Notebook or Jupyter Lab.

Open the file interactive_translation.ipynb.

Enter a sentence in English or Icelandic, and view the translation in real-time.

File Structure

train_model.py: Contains code for data preprocessing, model training, and saving.

evaluate_model.py: Evaluates model performance using perplexity.

interactive_translation.ipynb: Interactive interface for testing translations.

requirements.txt: List of required Python packages.

trained_gpt2/: Directory to save trained model and tokenizer.

Key Parameters

Max Length: Maximum token length for inputs (default: 128).

Learning Rate: .

Batch Size: 4 (both training and validation).

Epochs: 10.

Beam Search: Used for generating translations, with a beam size of 5.

Future Improvements

Expand dataset to include additional language pairs.

Optimize the model for faster inference.

Integrate the application into a web-based interface.

Acknowledgements

Hugging Face for providing the GPT-2 model and libraries.

PyTorch for enabling seamless implementation and training.

License

This project is licensed under the MIT License. See the LICENSE file for details.