mT5-base-turkish-qa

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

mT5-base-turkish-qa

This model is a fine-tuned version of google/mt5-base on the ucsahin/TR-Extractive-QA-82K dataset. It achieves the following results on the evaluation set:

• Loss: 0.5109
• Rouge1: 79.3283
• Rouge2: 68.0845
• Rougel: 79.3474
• Rougelsum: 79.2937

Model description

mT5-base model is trained with manually curated Turkish dataset consisting of 65K training samples with ("question", "answer", "context") triplets.

Intended uses & limitations

The intended use of the model is extractive question answering.

In order to use the inference widget, enter your input in the format:

Soru: question_text
Metin: context_text

Generated response by the model:

Cevap: answer_text

Use with Transformers:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from datasets import load_dataset

# Load the dataset
qa_tr_datasets = load_dataset("ucsahin/TR-Extractive-QA-82K")

# Load model and tokenizer
model_checkpoint = "ucsahin/mT5-base-turkish-qa"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)

inference_dataset = qa_tr_datasets["test"].select(range(10))

for input in inference_dataset:
    input_question = "Soru: " + input["question"]
    input_context = "Metin: " + input["context"]

    tokenized_inputs = tokenizer(input_question, input_context, max_length=512, truncation=True, return_tensors="pt")
    outputs = model.generate(input_ids=tokenized_inputs["input_ids"], max_new_tokens=32)
    output_text = tokenizer.batch_decode(outputs, skip_special_tokens=True)

    print(f"Reference answer: {input['answer']}, Model Answer: {output_text}")

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1

Training results

Framework versions

• Transformers 4.36.2
• Pytorch 2.1.0+cu118
• Datasets 2.16.1
• Tokenizers 0.15.0