wmt22-comet-da-paragraph-biomqm

wmt22-comet-da-paragraph-biomqm

Unbabel/wmt22-comet-da continued on paragraph-level data — a reference-based COMET metric fine-tuned on multi-sentence translation windows, for research on how learned MT metrics behave on inputs longer than one sentence.

Training

• Base: Unbabel/wmt22-comet-da (XLM-R large encoder, regression head).
• Data: AdleBenSalem/bio-mqm-paragraphs — length-balanced windows of k ∈ {1, 2, 3, 4, 6} consecutive segments of one document from Bio-MQM (biomedical MQM annotations; Zouhar et al. 2024), 8 directions: en↔de/es/fr/ru. Every window length contributes equally.
• Target: per-window mean segment-level MQM penalty, z-normalised and sigmoid-squashed per language pair (fit on train).
• Schedule: default COMET continue-training (encoder unfrozen after 30 % of the first epoch, embeddings kept frozen); max_epochs unbounded with early stopping on validation Kendall τ (patience 20). The run trained 25 epochs and this checkpoint is the best one (epoch 5, val Kendall τ = 0.362 on the held-out paragraph windows, all k pooled).

Usage

from huggingface_hub import snapshot_download
from comet import load_from_checkpoint   # pip install unbabel-comet

path = snapshot_download("AdleBenSalem/wmt22-comet-da-paragraph-biomqm")
model = load_from_checkpoint(f"{path}/checkpoints/model.ckpt")

data = [{"src": "...", "mt": "...", "ref": "..."}]   # sentences OR paragraphs
print(model.predict(data, batch_size=8, gpus=1).scores)

Intended use & caveats

• Built for research on length behaviour of learned MT metrics, not as a general-purpose replacement for wmt22-comet-da.
• Score scale differs from the base model: outputs live in (0, 1) on a per-language-pair normalised MQM-penalty scale (higher = better). Scores are not comparable to DA-trained COMET scores, nor across language pairs.
• Training data is biomedical; behaviour on other domains inherits from the base model plus this domain shift.
• Inputs beyond ~512 XLM-R tokens are truncated by the encoder, like every XLM-R-based COMET.

Citation

Please cite COMET and Bio-MQM:

@inproceedings{rei-etal-2022-comet,
  title     = "{COMET}-22: Unbabel-{IST} 2022 Submission for the Metrics Shared Task",
  author    = "Rei, Ricardo and C. de Souza, Jos{\'e} G. and Alves, Duarte and
               Zerva, Chrysoula and Farinha, Ana C and Glushkova, Taisiya and
               Lavie, Alon and Coheur, Luisa and Martins, Andr{\'e} F. T.",
  booktitle = "Proceedings of the Seventh Conference on Machine Translation (WMT)",
  year      = "2022"
}
@inproceedings{zouhar-etal-2024-fine,
  title     = "Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains",
  author    = "Zouhar, Vil{\'e}m and Ding, Shuoyang and Currey, Anna and
               Badeka, Tatyana and Wang, Jenyuan and Thompson, Brian",
  booktitle = "Proceedings of ACL 2024 (Short Papers)",
  year      = "2024",
  note      = "arXiv:2402.18747"
}