roberta-large-question-classifier

roberta-large-question-classifier

This model classifies questions according to the question-type ontology defined in following paper: Controllable Open-ended Question Generation with A New Question Type Ontology (Cao & Wang, ACL-IJCNLP 2021). It is a fine-tuned roberta-large on the open_question_type dataset. It achieves the following results on the test set:

              precision    recall  f1-score   support
       cause       0.91      0.93      0.92        91
  comparison       0.62      0.83      0.71        30
     concept       0.85      0.65      0.74        54
 consequence       0.80      0.73      0.76        11
 disjunction       0.80      0.78      0.79        36
     example       0.83      0.85      0.84       139
      extent       0.82      0.94      0.87        48
  judgmental       0.68      0.56      0.62        94
  procedural       0.86      0.88      0.87        85
verification       0.79      0.86      0.83        72
    accuracy                           0.81       660
   macro avg       0.80      0.80      0.80       660
weighted avg       0.81      0.81      0.81       660

Training procedure

Script: https://gist.github.com/jantrienes/329479bdad6b2a239cfcea83b9159a8a

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 512
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 30

Training results

Framework versions

• Transformers 4.33.2
• Pytorch 2.1.0+cu118
• Datasets 2.14.5
• Tokenizers 0.13.3