PreCOMET-disc

PreCOMET-disc

This is a source-only COMET model used for efficient evaluation subset selection. Specifically this model predicts discriminability distilled from an IRT model from up to WMT2022 (inclusive). The lower the scores, the better it is for evaluation because it will distinguish between two models of similar quality. It is not compatible with the original Unbabel's COMET and to run it you have to install github.com/zouharvi/PreCOMET:

pip install pip3 install git+https://github.com/zouharvi/PreCOMET.git

You can then use it in Python:

import precomet
model = precomet.load_from_checkpoint(precomet.download_model("zouharvi/PreCOMET-disc"))
model.predict([
  {"src": "This is an easy source sentence."},
  {"src": "this is a much more complicated source sen-tence that will pro·bably lead to loww scores 🤪"}
])["scores"]
> [1.4137403964996338, 0.6074056625366211]

The primary use of this model is from the subset2evaluate package:

import subset2evaluate

data_full = subset2evaluate.utils.load_data("wmt23/en-cs")
data_random = subset2evaluate.select_subset.basic(data_full, method="random")
subset2evaluate.evaluate.eval_subset_clusters(data_random[:100])
> 1
subset2evaluate.evaluate.eval_subset_correlation(data_random[:100], data_full)
> 0.71

Random selection gives us only one cluster and system-level Spearman correlation of 0.71 when we have a budget for only 100 segments. However, by using this model:

data_precomet = subset2evaluate.select_subset.basic(data_full, method="precomet_disc")
subset2evaluate.evaluate.eval_subset_clusters(data_precomet[:100])
> 1
subset2evaluate.evaluate.eval_subset_correlation(data_precomet[:100], data_full)
> 0.75

we get higher correlation. Note that this is not the best PreCOMET model and you can expect a bigger effect on a larger scale, as described in the paper.

This work is described in How to Select Datapoints for Efficient Human Evaluation of NLG Models?. Cite as:

@misc{zouhar2025selectdatapointsefficienthuman,
    title={How to Select Datapoints for Efficient Human Evaluation of NLG Models?}, 
    author={Vilém Zouhar and Peng Cui and Mrinmaya Sachan},
    year={2025},
    eprint={2501.18251},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2501.18251}, 
}