CJK-Phi-3.5-tokenizer

Phi-3.5-mini-instruct Tokenizer supporting Simplified Chinese, Traditional Chinese, Japanese, and Korean.

Example

from transformers import LlamaTokenizer

base_tokenizer = LlamaTokenizer.from_pretrained("microsoft/Phi-3.5-mini-instruct")
new_tokenizer = LlamaTokenizer.from_pretrained("nntsuzu/CJK-Phi-3.5-tokenizer")

text = (
    "Hello. This sentence is written in English." \
    "你好。这句话是用简体中文写的。" \
    "你好。這句話是用繁體中文寫的。" \
    "こんにちは。これは日本語で書かれた文章です。" \
    "안녕하세요. 이 문장은 한국어로 쓰여졌습니다."
)

print("----------- Tokenzier test -----------")
print(f"Phi_3.5_mini_instruct: {base_tokenizer.tokenize(text)}\n")
print(f"new tokenizer: {new_tokenizer.tokenize(text)}\n")

This tokenizer supports many Traditional Chinese and Korean characters, so it can likely be used for Korean mixed scripts as well.

Example

from transformers import LlamaTokenizer

base_tokenizer = LlamaTokenizer.from_pretrained("microsoft/Phi-3.5-mini-instruct")
new_tokenizer = LlamaTokenizer.from_pretrained("nntsuzu/CJK-Phi-3.5-tokenizer")

text = "이 토크나이저는 많은 繁体字나 한글을 支援하기 때문에 한글과 漢字가 混用한 文章에도 使用할 수 있는 可能性이 있습니다."

print("----------- Tokenzier test -----------")
print(f"Phi_3.5_mini_instruct: {base_tokenizer.tokenize(text)}\n")
print(f"new tokenizer: {new_tokenizer.tokenize(text)}\n")