GPT-2

Georgian Language Question-Answering Model

Project Overview

This project fine-tunes a pre-trained GPT-2 model for a Georgian language question-answering task. The model generates answers to questions in Georgian using a dataset containing question-answer pairs. The fine-tuned GPT-2 model is evaluated using common NLP metrics such as BLEU and ROUGE. The trained model can be used for automatic question answering in Georgian.

Key Features

• Fine-tuned GPT-2 model for Georgian question-answering.
• Trained on the MrDimitri/georgian-quora-faq dataset.
• Evaluation using metrics such as BLEU and ROUGE.
• Predictions are generated for test samples and compared against ground truth answers.

Requirements

• Python 3.6+
• PyTorch 1.9+ (for GPU support)
• Hugging Face transformers and datasets library
• Google Colab (recommended for GPU access)
• SacreBLEU and Rouge-Score libraries for evaluation metrics.

Dataset

The dataset used for training is MrDimitri/georgian-quora-faq from Hugging Face. It contains a collection of question-answer pairs in Georgian, ideal for training a question-answering model.

Dataset Structure

The dataset consists of two columns:

• question: The input question.
• answer: The expected answer.

Model

The model architecture is based on GPT-2, a Causal Language Model. It is a transformer-based model that is fine-tuned for the task of question answering by generating answers in the Georgian language.

Model Details

• Pre-trained Model: gpt2 (GPT-2 for general language modeling).
• Tokenizer: The model uses the GPT-2 tokenizer to preprocess input text.
• Special Tokens:
  • pad_token_id: Padding token ID to handle sequences of different lengths.
  • eos_token_id: End-of-sequence token used for sequence generation.

Training Process

Data Preprocessing

• Tokenization of input (question) and output (answer) sequences.
• Padding and truncation to a fixed length for model input.

Model Fine-Tuning

• The model is trained for 100 epochs.
• A batch size of 16 for both training and evaluation.
• Loss function: Cross-entropy loss for sequence generation.

Training Arguments

• evaluation_strategy: Evaluates the model at the end of each epoch.
• num_train_epochs: Number of training epochs (100).
• save_strategy: Saves the model at the end of each epoch.
• learning_rate: Set to 5e-5.
• gradient_accumulation_steps: Set to 2 for memory efficiency.

Evaluation

Metrics

BLEU Score

A metric for evaluating text generation in machine translation and question answering tasks.

ROUGE Scores

• ROUGE-1: Measures the overlap of unigrams.
• ROUGE-2: Measures the overlap of bigrams.
• ROUGE-L: Measures the longest common subsequence overlap.