Downloads · 30 days
0
TeXBLEU-Tokenizer
Kyudan/TeXBLEU-Tokenizer
TeXBLEU-Tokenizer is a machine learning model from Kyudan. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
from huggingfacehub import snapshotdownload
Access
Public
Updated Aug 30, 2024
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1 MB · 100%
At a glance
- Access
- Public
- Created
- Aug 29, 2024
- Updated
- Aug 30, 2024
- SHA
- 6c0177c9
Model card
From the Hugging Face model README
from huggingface_hub import snapshot_download
def load_tokenizer_from_hub(repo_name: str = "Kyudan/TeXBLEU-Tokenizer") -> Tokenizer: # Hugging Face Hub에서 토크나이저 다운로드 snapshot_dir = snapshot_download(repo_id=repo_name)
# 다운로드한 토크나이저 불러오기
tokenizer = Tokenizer.from_file(f"{snapshot_dir}/tokenizer.json")
return tokenizer
- Created
- Aug 29, 2024
- Updated
- Aug 30, 2024