Downloads · 30 days
14
15% of all-time downloads
pyctokenizer
raptorkwok/pyctokenizer
pyctokenizer is a translation model from raptorkwok. Use it when you need text moved from one language to another. The card lists the license as apache-2.0.
This is a Cantonese sentence tokenizer based on BART Chinese. It can be used along with our CCPC Parallel Corpus dataset.
All-time downloads
95
Public
Parameters
225M
939 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors899 MB · 95%
At a glance
- Task
- Translation
- License
- apache-2.0
- Model type
- bart
- Access
- Public
- Created
- Jan 20, 2026
- Updated
- Jan 20, 2026
- SHA
- fe84634b
Try a prompt
Base models
Model card
From the Hugging Face model README
Cantonese Tokenizer
This is a Cantonese sentence tokenizer based on BART Chinese. It can be used along with our CCPC Parallel Corpus dataset.
- Token Count: 161,278 (including 110,007 multi-character Chinese vocabularies)
Usage
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("raptorkwok/pyctokenizer")
print(tokenizer.tokenize("我哋去咗尖沙咀睇醫生呀！"))
# Output: ['我哋', '去', '咗', '尖沙咀', '睇醫生', '呀', '！']
- Task
- Translation
- Type
- bart
- License
- apache-2.0
- Languages
- zh
- Created
- Jan 20, 2026
- Updated
- Jan 20, 2026