coherence-all-mpnet-base-v2

Model Card for Coherence Testing Model

Model Details

Model Description

This model is a fine-tuned version of the sentence-transformers/all-mpnet-base-v2 designed specifically for coherence testing in dialogues. Leveraging the cross-encoder architecture from the sentence-transformers library, it is intended to evaluate the relevance and coherence of responses given a prompt or question.

• Developed by: Enoch Levandovsky
• Model type: Cross-encoder
• Language(s): English
• License: Check the repository for more information
• Finetuned from model: sentence-transformers/all-mpnet-base-v2

Model Sources

• Repository: Model on Hugging Face
• Space Demo: Coherence Testing Space

Uses

Direct Use

This model is designed to evaluate the coherence of a response to a given question or prompt. It can be directly used to enhance chatbots or dialogue systems by predicting how coherent or relevant a response is, thus improving the quality of conversational agents.

Downstream Use

This model can be fine-tuned further for specific dialogue systems or used as a component in larger conversational AI frameworks to ensure responses are meaningful and contextually appropriate.

Out-of-Scope Use

This model is not intended for applications requiring complex sentiment analysis, emotional tone recognition, or tasks outside dialogue coherence assessment.

How to Get Started with the Model

You can use the model as follows:

from sentence_transformers import CrossEncoder
model = CrossEncoder('enochlev/coherence-all-mpnet-base-v2')
output = model.predict([["What is your favorite color?", "Blue!"],
                        ["Do you like playing outside?", "I like ice cream."],
                        ["What is your favorite animal?", "I like dogs!"],
                        ["Do you want to go to the park?", "Yes, I want to go on the swings!"],
                        ["What is your favorite food?", "I like playing with blocks."],
                        ["Do you have a pet?", "Yes, I have a cat named Whiskers."],
                        ["What is your favorite thing to do on a sunny day?", "I like playing soccer with my friends."]])

print(output)

The output array represents coherence scores where higher scores indicate greater coherence.

Results

Example outputs reflect coherent or relevant responses with scores closer to 1. For instance:

Output >>> array([0.88097143, 0.04521223, 0.943173  , 0.9436357 , 0.04369843,
                  0.94450355, 0.8392763 ], dtype=float32)

Evaluation & Limitations

Testing Data, Factors & Metrics

The model has been fine-tuned and evaluated using the CHILDES dataset to ensure it captures conversational coherence effectively.

Recommendations

Users should be aware that while the model predicts coherence, it may not fully capture nuanced conversational elements such as sarcasm or humor.

Environmental Impact

Please refer to the Machine Learning Impact calculator for estimating carbon emissions. Details specific to training this model are not available but consider general best practices to minimize environmental impact.

Citation

To cite this model, please provide appropriate credit to the Hugging Face repository page and the original model creator, Enoch Levandovsky.