fake_news_detection_xlmRoberta

Write up:

Link to hugging face model:

https://huggingface.co/Sajib-006/fake_news_detection_xlmRoberta

Model Description:

* Used pretrained XLM-Roberta base model.
* Added classifier layer after bert model
* For tokenization, i used max length of text as 512(which is max bert can handle)

Result:

* Using bert base uncased english model, the accuracy was near 85% (For all samples)
* Using XLM Roberta base model, the accuracy was almost 100% ( For only 2k samples)

Limitations:

* Pretrained XLM Roberta is a heavy model. Training it with the full dataset(44k+ samples) was not possible using google colab free version. So i had to take small sample of 2k size for my experiment.
* As we can see, there is almost 100% accuracy and F1-score for 2000 dataset, so i haven't tried to find misclassified data.
* I couldn't run the model for the whole dataset as i used google colab free version, there was RAM and disk restrictions. XLMRoberta is a heavy model, so training it for the full dataset tends to take huge time. Colab doesn't provide GPU for long time. 
* As one run for one epoch took huge time, i had to save checkpoint after 1 epoch and retrain the model loading weights for 2nd time. After 2 epoch it showed almost 100% accuracy, so i didn't continue to train again.
* A more clear picture could have been seen if it could be run for the full dataset. I thought of some ideas about better model but couldn't implement for hardware restriction as mentioned and time constraint. My ideas are given below.

Ideas to imrove on full dataset:

*   Using XLM Roberta large instead of base can improve 
*   Adding dense layer and dropout layer to reduce overfitting(Though in my result there is 100% accuracy on hold-out test set, so no overfitting seems to be there)
*   Adding convolutional layer after the bert encoder work even better.
*   Combination of different complex convolution layers can be added to check if accuracy increases further more.
*   Hyperparameter tuning of the layers to ensure best result.