tf-tpu

unigram-tokenizer-wikitext

tf-tpu/unigram-tokenizer-wikitext

unigram-tokenizer-wikitext is a machine learning model from tf-tpu. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

This is a Unigram tokenizer trained on the Wikitext dataset. Refer to the trainunigram.py script within this repository to know how it was trained.