distillbert_conv_quality_score

distillbert_conv_quality_score

This model is a fine-tuned version of distilbert-base-uncased on the conv_ai_2 dataset. It was trained to generate a score (in the [0, 1] range) from a conversation

It achieves the following results on the evaluation set:

• training/loss: 0.0165
• validation/loss: 0.0149

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "alespalla/distillbert_conv_quality_score"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

conversation = '''
Q: Begin
A: lol ! do you think it is strange to feel like you have been through life before ?
Q: Hellow
A: I don't understand you 🙈. Also, try to guess: i like to ...
Q: How are you?
A: make time stop, funny you :)
Q: What is your name?
A: jessie. hows your day going ? 😃
'''

score = model(**tokenizer(conversation, return_tensors='pt')).logits.item()
print(f"Score: {score}")

Training and evaluation data

The training data was generated from conv_ai_2 using the following function

from datasets import load_dataset

def get_dataset(regression=False):

    db = load_dataset("conv_ai_2")

    def generate_converation(elem):
        text = ""
        for idx, txt in enumerate(elem["dialog"]):
            if idx % 2:
                text += f"A: {txt['text']}\n"
            else:
                text += f"Q: {txt['text']}\n"
        if regression:
            return {'text': text, "labels": (elem['eval_score'] - 1)/4}
        return {'text': text, "labels": elem['eval_score'] - 1}

    db = db.filter(lambda example: example["eval_score"] > 0)
    db = db.map(generate_converation, remove_columns=db['train'].column_names)
    db = db['train'].train_test_split(test_size=0.2).shuffle(42)

    return db

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• epochs: 40
• batch_size: 16
• learning_rate: 0.0002
• eval_steps: 82
• log_steps: 82
• save_steps: 41
• gradient_accumulation_steps: 1
• warmup_steps: 0

Training results

Framework versions

• Transformers 4.26.1
• Datasets 2.10.1
• Tokenizers 0.13.2