Downloads · 30 days
0
vietnamese-gpt2-tokenizer
B2306575/vietnamese-gpt2-tokenizer
vietnamese-gpt2-tokenizer is a machine learning model from B2306575. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
A ByteLevel BPE tokenizer trained on the wheevu/ct219-vietnamese-raw-400k Vietnamese corpus.
Access
Public
Updated Aug 14, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json2.1 MB · 100%
At a glance
- Access
- Public
- Created
- Aug 14, 2026
- Updated
- Aug 14, 2026
- SHA
- 28319b88
Model card
From the Hugging Face model README
Vietnamese GPT-2 Tokenizer
A ByteLevel BPE tokenizer trained on the wheevu/ct219-vietnamese-raw-400k Vietnamese corpus.
Configuration
- Model: BPE
- Vocabulary size: 30,000
- Pre-tokenizer: ByteLevel
- Decoder: ByteLevel
- Minimum frequency: 2
- Language: Vietnamese
Purpose
This tokenizer is designed to improve Vietnamese text tokenization for GPT-2 fine-tuning.
The original GPT-2 tokenizer produces inefficient tokenization for Vietnamese text because GPT-2 was originally trained primarily on non-Vietnamese text.
This tokenizer is trained specifically on Vietnamese text.
Training data
Dataset: wheevu/ct219-vietnamese-raw-400k
- Created
- Aug 14, 2026
- Updated
- Aug 14, 2026