ssa-comet-mtl

SSA-COMET-MTL, a robust, unified automatic metric for both MTE and QE, built based on SSA-MTE: It receives a triplet with (source sentence, translation, reference translation) for MTE, or a pair with (source sentence, translation) for QE, and returns a score that reflects the quality of the translation. This model is based on an improved African enhanced encoder, afro-xlmr-large-76L.

Paper

Li S., Wang J., Ali F., Cherry C., Deutsch D., Briakou E., Sousa-Silva R., Cardoso H.L, Stenetorp P., and Adelani D.I. SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?. In EMNLP 2025

License

Apache-2.0

Usage (SSA-COMET)

Using this model requires unbabel-comet to be installed:

pip install --upgrade pip  # ensures that pip is current 
pip install unbabel-comet

Then you can use it through comet CLI:

comet-score -s {source-inputs}.txt -t {translation-outputs}.txt -r {references}.txt --model McGill-NLP/ssa-comet-mtl

Or using Python:

from comet import download_model, load_from_checkpoint
model_path = download_model("McGill-NLP/ssa-comet-mtl")
model = load_from_checkpoint(model_path)
data = [
    {
        "src": "Nadal sàkọọ́lẹ̀ ìforígbárí o ní àmì méje sóódo pẹ̀lú ilẹ̀ Canada.",
        "mt": "Nadal's head to head record against the Canadian is 7–2.",
        "ref": "Nadal scored seven unanswered points against Canada."
    },
    {
        "src": "Laipe yi o padanu si Raoniki ni ere Sisi Brisbeni.",
        "mt": "He recently lost against Raonic in the Brisbane Open.",
        "ref": "He recently lost to Raoniki in the game Sisi Brisbeni."
    }
]
model_output = model.predict(data, batch_size=8, gpus=1)
print (model_output)

Intended uses

Our model is intended to be used for MT evaluation and Quality Eestimation.

Given a triplet with (source sentence, translation, reference translation), or a pair with (source sentence, translation), it outputs a single score between 0 and 1, where 1 represents a perfect translation.

Languages Covered:

There are 76 languages available :

• English (eng)
• Amharic (amh)
• Arabic (ara)
• Somali (som)
• Kiswahili (swa)
• Portuguese (por)
• Afrikaans (afr)
• French (fra)
• isiZulu (zul)
• Malagasy (mlg)
• Hausa (hau)
• chiShona (sna)
• Egyptian Arabic (arz)
• Chichewa (nya)
• Igbo (ibo)
• isiXhosa (xho)
• Yorùbá (yor)
• Sesotho (sot)
• Kinyarwanda (kin)
• Tigrinya (tir)
• Tsonga (tso)
• Oromo (orm)
• Rundi (run)
• Northern Sotho (nso)
• Ewe (ewe)
• Lingala (lin)
• Twi (twi)
• Nigerian Pidgin (pcm)
• Ga (gaa)
• Lozi (loz)
• Luganda (lug)
• Gun (guw)
• Bemba (bem)
• Efik (efi)
• Luvale (lue)
• Luba-Lulua (lua)
• Tonga (toi)
• Tshivenḓa (ven)
• Tumbuka (tum)
• Tetela (tll)
• Isoko (iso)
• Kaonde (kqn)
• Zande (zne)
• Umbundu (umb)
• Mossi (mos)
• Tiv (tiv)
• Luba-Katanga (lub)
• Fula (fuv)
• San Salvador Kongo (kwy)
• Baoulé (bci)
• Ruund (rnd)
• Luo (luo)
• Wolaitta (wal)
• Swazi (ssw)
• Lunda (lun)
• Wolof (wol)
• Nyaneka (nyk)
• Kwanyama (kua)
• Kikuyu (kik)
• Fon (fon)
• Bambara (bam)
• Chokwe (cjk)
• Dinka (dik)
• Dyula (dyu)
• Kabyle (kab)
• Kamba (kam)
• Kabiyè (kbp)
• Kanuri (knc)
• Kimbundu (kmb)
• Kikongo (kon)
• Nuer (nus)
• Sango (sag)
• Tamasheq (taq)
• Tamazight (tzm)
• N'ko (nqo)

Specifically Finetuned on:

• Amharic (amh)
• Hausa (hau)
• Igbo (ibo)
• Kikuyu (kik)
• Kinyarwanda (kin)
• Luo (luo)
• Twi (twi)
• Yoruba (yor)
• Zulu (zul)
• Ewe (Ewe)
• Lingala (lin)
• Wolof (wol)