lcw99

tokenizer-gpt-j-ext-ko

lcw99/tokenizer-gpt-j-ext-ko

tokenizer-gpt-j-ext-ko is a machine learning model from lcw99. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

This tokenizer is based on EleutherAI/gpt-j-6B and increases the vocabulary size to 91238 by adding Korean words.