distilbert-squad-qa

distilbert-base-uncased fine-tuned on SQuAD

This model is a fine-tuned version of distilbert-base-uncased on the SQuAD dataset.

Training Details

Training Hyperparameters

• Model: distilbert-base-uncased
• Dataset: SQuAD
• Optimizer: adamw
• Learning Rate Scheduler: cosine_with_restarts
• Learning Rate: 3e-05
• Batch Size: 16 per device
• Total Batch Size: 64
• Epochs: 5 (with early stopping)
• Weight Decay: 0.01
• Warmup Ratio: 0.06
• Max Gradient Norm: 1.0

Early Stopping

• Patience: 4
• Metric: f1
• Best Epoch: 3

Usage

from transformers import AutoTokenizer, AutoModelForQuestionAnswering

tokenizer = AutoTokenizer.from_pretrained("HariomSahu/distilbert-squad-qa")
model = AutoModelForQuestionAnswering.from_pretrained("HariomSahu/distilbert-squad-qa")

# Example usage
question = "What is the capital of France?"
context = "France is a country in Europe. Its capital city is Paris."

inputs = tokenizer(question, context, return_tensors="pt")
outputs = model(**inputs)

# Get answer
start_scores, end_scores = outputs.start_logits, outputs.end_logits
start_index = start_scores.argmax()
end_index = end_scores.argmax()
answer = tokenizer.decode(inputs["input_ids"][0][start_index:end_index+1])
print(f"Answer: {answer}")

Evaluation Results

The model achieved the following results on the evaluation set:

• Exact Match: 76.9253
• F1 Score: 85.2786

Training Configuration Hash

Config Hash: fe08f7bd

This hash can be used to reproduce the exact training configuration.