scaling-qa

'scaling-qa' is a dataset made of merging a wide corpus on existing Question Answering datasets.

The dataset has been created with the sole purpose of augmenting Text-to-Text transformer-based generation models like T5, and has shown a great performance increase so far.

The dataset is in SQuAD format, and includes 3 fields: 'context', 'question' and 'answer'. In the dataset, there are many questions and answers over the same context, which adds up to the models' abilities to generate diverse questions or answers.

The dataset can be used for Question Answering (QA) or Question Generation (QG) tasks.

A pipeline for generating diverse questions is currently under preparation.

This dataset is a collection of: SQuAD, NewsQA, DROP, Duo_RC and COQA datasets. The conversational question-answer pairs in the COQA dataset have been made into separate entries, with multiple question and answer pairs corresponding to the same context. This structure helps the models understand this conversational behaviour when needed, but prevents them from overfitting into a strictly conversational model.

All licences originally belong to the owners of mentioned datasets, this is an effort to create a large corpus for scaled models.

June 26, 2024 - Wednesday Serhan YILMAZ