Downloads · 30 days
0
Nepali_BPE_Tokenizer
raygx/Nepali_BPE_Tokenizer
Nepali_BPE_Tokenizer is a machine learning model from raygx. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
- This tokenizer is a PreTrainedTokenizerFast which is trained on raygx/Nepali-Extended-Corpus datasets. - This tokenizer is trained from scratch using Tokenizers library. - This tokenizer uses - Model: BPE(unktoken="…
Access
Public
Updated Feb 21, 2024
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1.3 MB · 100%
At a glance
- Access
- Public
- Created
- Jul 11, 2023
- Updated
- Feb 21, 2024
- SHA
- a8c09961
Model card
From the Hugging Face model README
This is only a tokenizer.
- This tokenizer is a PreTrainedTokenizerFast which is trained on raygx/Nepali-Extended-Corpus datasets.
- This tokenizer is trained from scratch using Tokenizers library.
- This tokenizer uses
- Model: BPE(unk_token="[UNK]")
- Normalizer: normalizers.Sequence([NFD(),Strip()])
- Pre-processor: pre_tokenizers.Sequence([Whitespace(),Digits(individual_digits=True), Punctuation()])
- Post-processor: BertProcessing
- Created
- Jul 11, 2023
- Updated
- Feb 21, 2024