bilstm-evidence-detection

language: en license: cc-by-4.0 tags:

• pairwise-sequence-classification-evidence-detection repo: https://github.com/RibhavOjha/NLU_ed

---

Model Card for t56225ro-p37429am-ED

<!-- Provide a quick summary of what the model is/does. -->

This is a classification model that was trained to detect, given a pair of claim and evidence, if the evidence supports or refutes the claim.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This model is based upon a BiLSTM model that was fine-tuned on 23.7K pairs of texts as a part of the ED (Evidence Detection) dataset.

This model is intended for the task of pairwise sequence classification for ED. It can be further finetuned on related pairwise sequence classification tasks. This model is inspired from the model discussed in in the paper "BiLSTM-Attention and LSTM-Attention via Soft Voting in Emotion Classification". The link can be found below.

• Developed by: Ribhav Ojha and Amal Manzoor
• Language(s): English
• Model type: Supervised
• Model architecture: BiLSTM

Model Resources

<!-- Provide links where applicable. -->

• Paper or documentation: https://www.researchgate.net/publication/329512919_NLP_at_IEST_2018_BiLSTM-Attention_and_LSTM-Attention_via_Soft_Voting_in_Emotion_Classification

Training Details

Training Data

<!-- This is a short stub of information on the training data that was used, and documentation related to data pre-processing or additional filtering (if applicable). -->

23702 pairs of texts given in the coursework

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

The input claims and evidences are first converted into word embeddings using keras. Each of the embeddings is passed through a BiLSTM layer and then to a dropout layer. The output is concatenated and passed through an attention layer. The output is then merged and again passed through a BiLSTM layer to finally be passed to the output layer. The model is trained f0r 1 epoch. with a batch size of 32. Different batch sizes, learning rates, and model architectures were experimented with. First, the training data was split into training and validation. This was done to keep track of model's performance while training. However, it was realised that combining to two to train the model yielded better performance. Next, the trained model was tested on the dev dataset. Finally, the predictions were generated using the test dataset.

Training Hyperparameters

<!-- This is a summary of the values of hyperparameters used in training the model. -->

  - learning_rate: 0.001
  - train_batch_size: 32
  - num_epochs: 1
  - optimizer: Adam
  - embedding_dim = 100
  - LSTM dense units: 64 & 32
  - Dropout Rate: 0.2
 

Speeds, Sizes, Times

<!-- This section provides information about how roughly how long it takes to train the model and the size of the resulting model. -->

The model was trained on a T4 GPU on Google Colab.

  - overall training time: 20 minutes
  - duration per training epoch: 20 minutes
  - model size: 417MB

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data & Metrics

Testing Data

<!-- This should describe any evaluation data used (e.g., the development/validation set provided). -->

A subset of the development set provided, amounting to 5926 pairs.

Metrics

<!-- These are the evaluation metrics being used. -->

  - Precision
  - Recall
  - F1-score
  - Accuracy
  - Support
  - Macro Average
  - Weighted Average
  

Results

The model obtained an F1-score of 87% for irrelvant pairs, and 65% for relevant pairs. The model obtained an accuracy of 81% on the dev dataset. The macro avg and weighted avg numbers are close to each other, suggesting that the model's performance is consistent across classes and considering the class distribution.

Technical Specifications

Hardware

  - GPU: T4

Software

  - Transformers 4.18.0
  - Pytorch 1.11.0+cu113

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

It is possible that the model is biased towards the small amount of training data it is trained on. It might perform poorly on data gathered from different sources.

Training data is limited in size. Increasing the size of the training data can help the model to generalise on unseen data better.

Additional Information

<!-- Any other information that would be useful for other people to know. -->

The hyperparameters were determined by experimentation with different values.