Neural_Network_Based_Language_Model_for_Next_Token_Prediction

Bilingual Language Model with BPE Tokenizer

Hey there! This is my cool project where I built a language model that can handle both English and Chinese. It's like teaching a computer to understand and generate text in two totally different languages!

What's in this project?

1. BPE Tokenizer: I made a special tool called a Byte Pair Encoding (BPE) tokenizer. It's like a smart dictionary that breaks words into smaller pieces, which helps the model understand language better.

2. LSTM Language Model: I used something called an LSTM (Long Short-Term Memory) neural network. It's a fancy way of saying the model can remember important stuff from earlier in a sentence to help predict what comes next.

3. Bilingual Support: The coolest part is that this model can work with both English and Chinese! It's like having a brain that can switch between two languages.

How it works

1. The BPE tokenizer learns to split words into subwords. For example, "playing" might become "play" + "ing".
2. The LSTM model then learns patterns in these subwords to predict what comes next in a sentence.
3. When we want to generate text, the model uses what it learned to create new sentences in either English or Chinese.

What I learned

• How to implement a BPE tokenizer from scratch (it was tricky but fun!)
• Training a neural network on a bilingual dataset
• Dealing with perplexity (a way to measure how good the model is at predicting text)
• Generating text in two very different languages

Results

My model achieved a perplexity score of 32.83, which is pretty good for a bilingual model! It means the model is getting better at predicting what word comes next in a sentence.

Next steps

I'm thinking of trying out more advanced models like Transformers to see if I can make it even better. Also, I want to experiment with more languages - maybe add Spanish or French to the mix!

Thanks for checking out my project! Feel free to play around with it and let me know what you think. 😊