vibrato-models

Vibrato Model Archive

This repository hosts all models from the Vibrato(url) GitHub release. Models that were compressed using zstd have already been decompressed, so they are ready for direct download and use.

Important: This repository deliberately does not provide a unified license because it contains models under different licensing terms. While most models have been detected to use the BSD license, some do not have a standard license. It is recommended to check the license file in each folder before using the models.

For those seeking models under the BSD license, you can visit the following repository (ryan-minato/vibrato-models-bsdonly).

Available Models

• bccwj-suw+unidic-cwj-3_1_1+compact-dual
• bccwj-suw+unidic-cwj-3_1_1+compact
• bccwj-suw+unidic-cwj-3_1_1-extracted+compact-dual
• bccwj-suw+unidic-cwj-3_1_1-extracted+compact
• bccwj-suw+unidic-cwj-3_1_1
• ipadic-mecab-2_7_0-small
• ipadic-mecab-2_7_0
• jumandic-mecab-7_0
• naist-jdic-mecab-0_6_3b
• unidic-cwj-3_1_1+compact-dual
• unidic-cwj-3_1_1+compact
• unidic-cwj-3_1_1
• unidic-mecab-2_1_2

Usage

from huggingface_hub import hf_hub_download
import vibrato

# Load tokenizer from `.cache/hf`
model_path = hf_hub_download("ryan-minato/vibrato-models", "<<model_name>>/system.dic")
with open(model_path, "rb") as f:
  tokenizer = vibrato.Vibrato(f.read())

text = """\
「四十二だと!」ルーンクォールが叫んだ。
「七百五十万年かけて、それだけか?」
「何度も徹底的に検算しました」コンピュータが応じた。
「まちがいなくそれが答えです。率直なところ、みなさんのほうで究極の疑問が何であるかわかっていなかったところに問題があるのです」
"""

tokenizer.tokenize(text)