sentinel-ref-mqm

<div align="center"> <h1 style="font-family: 'Arial', sans-serif; font-size: 28px; font-weight: bold; color: black;"> 🛡️ Guardians of the Machine Translation Meta-Evaluation:<br> Sentinel Metrics Fall In! </h1> </div> <div style="display:flex; justify-content: center; align-items: center; flex-direction: row;"> <a href="https://2024.aclweb.org/"><img src="http://img.shields.io/badge/ACL-2024-4b44ce.svg"></a> &nbsp; &nbsp; <a href="https://aclanthology.org/2024.acl-long.856/"><img src="http://img.shields.io/badge/paper-ACL--anthology-B31B1B.svg"></a> &nbsp; &nbsp; <a href="https://arxiv.org/abs/2408.13831"><img src="https://img.shields.io/badge/arXiv-2408.13831-b31b1b.svg"></a> &nbsp; &nbsp; <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg"></a> </div> <div style="display:flex; justify-content: center; align-items: center; flex-direction: row;"> <a href="https://huggingface.co/collections/sapienzanlp/mt-sentinel-metrics-66ab643b32aab06f3157e5c1"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-FCD21D"></a> &nbsp; &nbsp; <a href="https://github.com/SapienzaNLP/guardians-mt-eval"><img src="https://img.shields.io/badge/GitHub-Repo-121013?logo=github&logoColor=white"></a> &nbsp; &nbsp; </div>

This repository contains the SENTINEL_REF metric model pre-trained on Direct Assessments (DA) annotations and further fine-tuned on Multidimensional Quality Metrics (MQM) data. For details on how to use our sentinel metric models, check our GitHub repository.

Usage

After having installed our repository package, you can use this model within Python in the following way:

from sentinel_metric import download_model, load_from_checkpoint

model_path = download_model("sapienzanlp/sentinel-ref-mqm")
model = load_from_checkpoint(model_path)

data = [
    {"ref": "There's no place like home."},
    {"ref": "Toto, I've a feeling we're not in Kansas anymore."}
]

output = model.predict(data, batch_size=8, gpus=1)

Output:

# Segment scores
>>> output.scores
[0.5577929019927979, 0.3894208073616028]

# System score
>>> output.system_score
0.4736068546772003

Cite this work

This work has been published at ACL 2024 (Main Conference). If you use any part, please consider citing our paper as follows:

@inproceedings{perrella-etal-2024-guardians,
    title = "Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!",
    author = "Perrella, Stefano  and Proietti, Lorenzo  and Scir{\`e}, Alessandro  and Barba, Edoardo  and Navigli, Roberto",
    editor = "Ku, Lun-Wei  and Martins, Andre  and Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand", publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.856",
    pages = "16216--16244",
}

License

This work is licensed under Creative Commons Attribution-ShareAlike-NonCommercial 4.0.