madlad400-7b-mt-bt-ct2-int8_float16

madlad400-7b-mt-bt model for CTranslate2

The model is quantized version of the jbochi/madlad400-7b-mt-bt with int8_float16 quantization and can be used in CTranslate2.

madlad400 is a multilingual machine translation model based on the T5 architecture introduced by Google DeepMind, Google Research in Sep 2023. It was trained on 250 billion tokens covering over 450 languages using publicly available data. The paper is titled "MADLAD-400: A Multilingual And Document-Level Large Audited Dataset" (arXiv:2309.04662).

madlad400-7b-mt-bt is finetuned version of the 7.2B parameter model on backtranslated data. Authors say in the paper that:

While this setup is very likely sub-optimal, we see that back-translation greatly improves en2xx translation (by 3.0 chrf, in the case of Flores-200) in most cases.

Conversion details

The original model was converted on 2023-12 with the following command:

ct2-transformers-converter --model jbochi/madlad400-7b-mt-bt --quantization int8_float16 --output_dir madlad400-7b-mt-bt-ct2-int8_float16 \
    --copy_files added_tokens.json generation_config.json model.safetensors.index.json shared_vocabulary.json special_tokens_map.json spiece.model tokenizer.json tokenizer_config.json

Example

This example code is obtained from CTranslate2_transformers. More detailed information about the translate_batch methon can be found at CTranslate2_Translator.translate_batch.

import ctranslate2
import transformers

translator = ctranslate2.Translator("avans06/madlad400-7b-mt-bt-ct2-int8_float16", compute_type="auto")
tokenizer = transformers.AutoTokenizer.from_pretrained("jbochi/madlad400-7b-mt-bt")

prefix = "<2zh> "
input_text = "Who is Alan Turing?"
input_tokens  = tokenizer.convert_ids_to_tokens(tokenizer.encode(prefix + input_text))

results = translator.translate_batch([input_tokens])

output_tokens = results[0].hypotheses[0]
output_text = tokenizer.decode(tokenizer.convert_tokens_to_ids(output_tokens))