bert-mini-fine-tuning-squad

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This modelcard aims to be a base template for new models. It has been generated using this raw template.

Task

Question-answer model in SQUAD dataset

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: Tô Hoàng Minh Tiến
• Finetuned from model : bert-mini

<!-- Provide the basic links for the model. -->

How to Get Started with the Model

Use the code below to get started with the model.

# Load model directly
from transformers import AutoTokenizer, TFAutoModelForQuestionAnswering

tokenizer = AutoTokenizer.from_pretrained("Tien-THM/bert-mini-fine-tuning-squad")
model = TFAutoModelForQuestionAnswering.from_pretrained("Tien-THM/bert-mini-fine-tuning-squad")

import numpy as np

def Inference(context, question):
  encoding = tokenizer(context, question, return_tensors='tf')
  start_pos = model(encoding).start_logits
  end_pos = model(encoding).end_logits
  s = np.argmax(start_pos[0])
  e = np.argmax(end_pos[0])
  print(tokenizer.decode(encoding['input_ids'][0][s:e+1]))

question = 'How many layes does BERT-large have'
context = 'BERT-large is really big... it has 24-layers and an embedding size 
of 1,024, for a total of 340M parameters! Altogether it is 1.34GB, so 
expect it to take a couple minutes to download to your Colab instance'

Inference(context, question)
# Answer: 24 - layers and an em ##bed ##ding size of 1 , 02 ##4

Training Details

Training Data

Using 2 datasets:

• SQUAD

Training Procedure

Optimization:

• Adam

Loss function

• Cross entropy

Training Hyperparameters

• Learning rate: 2e-5
• Batch size: 8
• Epoch: 4

Training Loss

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Metrics

• Exact Match: 0.45