madlad400-3b-mt-ct2-int8_float16

madlad400-3b-mt, CTranslate2 int8_float16

google/madlad400-3b-mt converted to the CTranslate2 format and quantised to int8_float16. The same model, 2.8 GB instead of 11.8 GB, and it loads without a conversion step.

The tokenizer files are included, so nothing else has to be downloaded.

Use

import ctranslate2
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

path = snapshot_download("olob0/madlad400-3b-mt-ct2-int8_float16")
tokenizer = AutoTokenizer.from_pretrained(path)
translator = ctranslate2.Translator(path, device="cuda", compute_type="int8_float16")

tokens = tokenizer.convert_ids_to_tokens(tokenizer("<2en> Bom dia.").input_ids)
result = translator.translate_batch([tokens])
print(tokenizer.decode(tokenizer.convert_tokens_to_ids(result[0].hypotheses[0])))

MADLAD takes the target language as a <2xx> token at the front of the source text. Put it in the string and let the tokenizer handle it. Prepending the bare token to the token list instead loses the word boundary piece in front of it, and the translation comes back mangled with no error.

How it was built

ct2-transformers-converter --model google/madlad400-3b-mt \
  --quantization int8_float16 --output_dir madlad400-3b-mt-ct2-int8_float16 \
  --copy_files tokenizer.json tokenizer_config.json special_tokens_map.json spiece.model added_tokens.json

License

Apache-2.0, inherited from the original model. Original work by Google, this is a format conversion of it.