ECoh-7B

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

ECoh is a family of transformer-based decoder-only language model finetuned to assess the coherence of responses in dialogue systems.

Model Details

Model Sources

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/johndmendonca/Ecoh
• Paper: https://arxiv.org/abs/2407.11660

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, AutoTokenizer

# load model
model_path="Johndfm/ECoh-7B"
tokenizer = AutoTokenizer.from_pretrained(model_path,padding_side="left")
base_model = AutoModelForCausalLM.from_pretrained(model_path).to("cuda")

# prepare example
example = "Context:\nA: Dahua's Market . How can I help you ? \nB:  Where is your store located ? \n\nResponse:\nA: Our store is located on 123 Main Street, in the city center."
messages = [
      {"role": "system", "content": "You are a Coherence evaluator."}
      {"role": "user", "content": f"{example}\n\nGiven the context, is the response Coherent (Yes/No)? Explain your reasoning."}
]

text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to("cuda")

generated_ids = base_model.generate(
        model_inputs.input_ids,
        max_new_tokens=64
)

generated_ids = [
        output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Training and Evaluation Details

Please refer to the original paper.

Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

@misc{mendonça2024ecoh,
      title={ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues}, 
      author={John Mendonça and Isabel Trancoso and Alon Lavie},
      year={2024},
      eprint={2407.11660},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.11660}, 
}

Model Card Contact

[email protected]