Parallel-T5-Translation-PyTorch

Parallel-T5-Translation-PyTorch

Project Title and Introduction

Parallel-T5-Translation-PyTorch is a custom, optimized Transformer-based sequence-to-sequence model inspired by T5-Small, developed for English-to-French Machine Translation.

The core innovation in this project is the Parallel Multi-Head Attention mechanism, designed to enable experimentation with model parallelism and improve attention efficiency. This implementation provides a foundation for studying how attention heads can be executed concurrently to enhance performance in translation tasks.

---

Custom Model Architecture: Parallel Attention

Overview

Our model, ParallelT5Small, replaces the standard Multi-Head Attention (MHA) with a novel Parallel Multi-Head Attention (P-MHA) layer.

• Standard MHA:
  Computes one set of Query (Q), Key (K), and Value (V) projections, then splits the resulting vectors across all heads.

• Parallel MHA (Proposed):
  Splits the attention mechanism into two parallel streams, each using separate Q/K/V projection weights for half of the attention heads.
  The results from both parallel streams are independently projected back to the hidden dimension and then summed to form the final attention output.

Goal

This architecture serves as a foundation for:

• Exploring architectural variants of the Transformer.
• Studying the effects of parallelized attention on translation performance.
• Investigating scalability in distributed training and efficiency on specialized hardware (e.g., GPUs or TPUs).

---

<h2>Model Architecture</h2> <p align="center"> <img src="Assets/Architecture_diagram.jpg" alt="Parallel T5 Architecture" width="100%"> </p>

---

<h2>Training & Evaluation Metrics (Epoch 37)</h2> <table> <thead> <tr> <th>Metric</th> <th>Train Result (Epoch 37)</th> <th>Validation Result (Epoch 37)</th> <th>Goal</th> </tr> </thead> <tbody> <tr> <td><strong>Loss (Cross-Entropy)</strong></td> <td>4.2213</td> <td>4.8907</td> <td>Decrease loss below <strong>2.0</strong></td> </tr> <tr> <td><strong>Token Accuracy</strong></td> <td>&#8776; 18.18%</td> <td>&#8776; 15.20%</td> <td>Achieve <strong>60%+</strong></td> </tr> <tr> <td><strong>BLEU Score</strong></td> <td><em>To be implemented</em></td> <td><em>To be implemented</em></td> <td>Target: <strong>30–40</strong></td> </tr> </tbody> </table>

---

Installation and Setup

Installation

To set up the project locally, follow these steps.
Python 3.8+ is required.

---

1Clone the Repository

git clone https://github.com/YourUsername/Parallel-T5-Translation-PyTorch.git
cd Parallel-T5-Translation-PyTorch

conda create -n parallel-t5 python=3.9
conda activate parallel-t5

# Install PyTorch (use the appropriate CUDA version for your setup)
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118

# Install project dependencies
pip install -r requirements.txt

Training and Preprocessing

The workflow consists of two main steps: data preparation and model training.

---

Step 1: Data Preprocessing

This step performs the following:

• Downloads the GlobalVoices EN-FR dataset
• Tokenizes data using the T5 tokenizer
• Splits into train, validation, and test sets
• Saves processed tensors to ./data/processed

Run the preprocessing:

python run.py

---

References

This project is built upon the foundational work of the T5 model and utilizes the publicly available GlobalVoices dataset.

🔹 T5 (Text-to-Text Transfer Transformer)

The model architecture is heavily inspired by the T5 framework, which casts all NLP problems into a text-to-text format.

Paper:
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019).

Link: https://arxiv.org/abs/1910.10683

🔹 OPUS - GlobalVoices Dataset

The parallel English-French data used for training is sourced from the OPUS collection’s GlobalVoices corpus.

Resource:
Jörg Tiedemann. The OPUS Parallel Corpus (2012).

Link (GlobalVoices source):
https://object.pouta.csc.fi/OPUS-GlobalVoices/v2018q4/moses/en-fr.txt.zip

🔹 Hugging Face Transformers Library

The AutoTokenizer and several best practices for transformer training and dataset handling are derived from the Hugging Face ecosystem.

Library:
https://huggingface.co/docs/transformers/index

---