next_token_prediction

Next Token Prediction Model

This project is a Next Token Prediction model built using PyTorch. The goal of this model is to predict the next word/token in a sequence of text, which is useful for tasks like text generation, autocompletion, and language modeling.

Model Overview

The model is designed to process sequences of text and predict the most likely token to follow a given sequence. It utilizes a neural network architecture with PyTorch for training and inference. The model can run on both CPU and GPU for efficient training and prediction.

Key Features

PyTorch-based: Built using PyTorch, making it flexible and easy to integrate with other PyTorch projects. Next Token Prediction: Designed specifically for predicting the next word/token in a text sequence. GPU Support: Can be trained and run on GPUs for improved performance. Installation To install the required dependencies, you can use the following command:

bash Copy code pip install torch torchvision Ensure you have PyTorch installed, and if you wish to use the GPU, install the appropriate CUDA version.

Usage

1. Clone the repository bash download code ipynb from raghavendra_midterm.ipynb cd next-token-prediction

2. Prepare the Data Ensure that your input text file is in the correct format. The model expects a text file named input.txt, which contains sequences of text.

3. Training the Model You can train the model using the provided Jupyter notebook:

bash Copy code jupyter notebook Raghavendra_midterm.ipynb The notebook includes steps for:

Importing necessary libraries. Loading and preprocessing the data. Setting up and training the model. 4. Running Inference. To use the model for next token prediction, you can run the inference code in the notebook. You can modify the following code to predict the next token for any given text input:.

python Copy code text = "Your input text here". predicted_token = model.predict_next_token(text). print(predicted_token). 5. GPU Support. Ensure that GPU support is enabled by checking the following:

python Copy code from raghvendra_midterm.ipynb. torch.cuda.is_available(). The model will automatically utilize the GPU if available.

Model Architecture The model uses a standard neural network architecture, including: Embedding layers for text input. 5 LSTM for sequence modeling. 5 Fully connected layers for prediction. You can modify the architecture in the notebook if needed.

You can found the different checkpoints of the model during training where checkpoint "checkpoint_step_450000.pth" is the best model with low loss for both train and Validation loss

Results

After training, the model should be able to predict the next token in a sequence with reasonable accuracy. The performance can be evaluated using metrics loss.