wmt21-dense-24-wide-en-x-st

facebook wmt21 model facebook/wmt21-dense-24-wide-en-x as safetensors for testing.

so far, looks quite good, cf. comet scores:

+-----------------------------------------+-----------------+
| File                                    |   Overall Score |
+=========================================+=================+
| Capybara_de_wmt21_scored.jsonl          |        0.848375 |
+-----------------------------------------+-----------------+
| Capybara_de_GPT4_scored.jsonl           |        0.846241 |
+-----------------------------------------+-----------------+
| Capybara_de_Claude-3-Opus_scored.jsonl  |        0.84568  |
+-----------------------------------------+-----------------+
| Capybara_de_deepl_scored.jsonl          |        0.843937 |
+-----------------------------------------+-----------------+
| Capybara_de_GPT3.5_scored.jsonl         |        0.843922 |
+-----------------------------------------+-----------------+
| Capybara_de_occiglot_scored.jsonl       |        0.83135  |
+-----------------------------------------+-----------------+
| Capybara_de_discolm_scored.jsonl        |        0.830676 |
+-----------------------------------------+-----------------+
| Capybara_de_nbbl_scored.jsonl           |        0.829132 |
+-----------------------------------------+-----------------+
| Capybara_de_wmt19_scored.jsonl          |        0.824847 |
+-----------------------------------------+-----------------+
| Capybara_de_t5madlad_scored.jsonl       |        0.818146 |
+-----------------------------------------+-----------------+
| Capybara_de_mixtral_scored.jsonl        |        0.807397 |
+-----------------------------------------+-----------------+
| Capybara_de_TowerInstruct2_scored.jsonl |        0.788971 |
+-----------------------------------------+-----------------+

also, cf. comparison on a few snippets: https://huggingface.co/spaces/cstr/compare_translations

regarding quantization: on linux, or windows wsl (with accelerate, triton), you can use quantized versions q8 q4

on apple mac mps, you can use CTranslate2 like this: first convert the model:

ct2-transformers-converter --model cstr/wmt21-dense-24-wide-en-x-st --quantization int8_float32 --output_dir wmt21ct2_int8

then run e.g. in python:

import ctranslate2
import transformers

translator = ctranslate2.Translator("wmt21ct2_int8")
tokenizer = transformers.AutoTokenizer.from_pretrained("facebook/wmt21-dense-24-wide-en-x")
tokenizer.src_lang = "en"

source = tokenizer.convert_ids_to_tokens(tokenizer.encode("Choose the correct verb form to complete the sentence: The birds ____________ (to fly) to the south for the winter."))
target_prefix = [tokenizer.lang_code_to_token["de"]]
results = translator.translate_batch([source], target_prefix=[target_prefix])
target = results[0].hypotheses[0][1:]

print(tokenizer.decode(tokenizer.convert_tokens_to_ids(target)))

Provenance and EU AI Act Art. 53 note

• Upstream model: facebook/wmt21-dense-24-wide-en-x — published by facebook.
• Upstream licence: mit. This repository redistributes under the same terms; it grants no rights the upstream licence does not.
• What was done here: format conversion and/or quantisation only (see the file list in this repository). No training, no fine-tuning, no merging, no distillation, no change to architecture, vocabulary or capability. Only the numeric representation of the upstream weights differs.
• Training data: documented — where it is documented at all — by the upstream provider; see the upstream model card. No training data was used, added or selected by this repository. No training-content summary was found on the upstream model card at the time of writing; that documentation gap is upstream's and is not filled here.
• Provider status: under Regulation (EU) 2024/1689 the upstream authors remain the provider of this model. Converting the serialisation format does not make this repository the provider of a new general-purpose AI model, and no such claim is made. Questions about training content, copyright policy or model capability belong upstream.