backpack-gpt2-nli

Model Card for Backpack-GPT2-NLI

This is a fine-tuned version of backpack-gpt2 with a NLI classification head on the esnli dataset. Results:

• On Validation Set:
  • CrossEntropyLoss: 0.3168
  • Accuracy: 0.9006
  • F1: 0.9004
• On Test Set:
  • CrossEntropyLoss: 0.3277
  • Accuracy: 0.8958
  • F1: 0.8955

Model Description

• Developed by: Erfan Moosavi Monazzah
• Model type: Sequence Classifier
• Language(s) (NLP): English
• License: apache-2.0
• Finetuned from model [optional]: Backpack-GPT2

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained('gpt2')
tokenizer.pad_token = tokenizer.eos_token

def tokenize_function(examples):
    concatenated_sentences = [f'{premise.strip(".")}. ^ {hypothesis.strip(".")}.' for premise, hypothesis in zip(examples['premise'], examples['hypothesis'])]

    tokenized_inputs = tokenizer(
        concatenated_sentences,
        padding="max_length",
        truncation=True,
        max_length=512,
        return_tensors="pt",
    )
    return tokenized_inputs

model = AutoModelForSequenceClassification.from_pretrained('ErfanMoosaviMonazzah/backpack-gpt2-nli', trust_remote_code=True)
model.eval()

tokenized_sent = tokenize_function({
    'premise':['A boy is jumping on skateboard in the middle of a red bridge.',
               'Two women who just had lunch hugging and saying goodbye.',
               'Children smiling and waving at camera'],
    'hypothesis':['The boy does a skateboarding trick.',
                  'The friends have just met for the first time in 20 years, and have had a great time catching up.',
                  'The kids are frowning']
})
model.predict(input_ids=tokenized_sent['input_ids'], attention_mask=tokenized_sent['attention_mask'])

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-5
• train_batch_size: 64
• eval_batch_size: 64
• seed: 2023
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0
• num_epochs: 3

Training results

Model Card Authors

Erfan Moosavi Monazzah