Downloads · 30 days
0
wikitext-wordlevel
dustalov/wikitext-wordlevel
wikitext-wordlevel is a machine learning model from dustalov. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for tokenizers. The card lists the license as cc-by-sa-3.0.
This is a simple word-level tokenizer created using the Tokenizers library. It was trained for educational purposes on the combined train, validation, and test splits of the WikiText-103 corpus.
Access
Public
Updated Jan 22, 2024
Repo size
—
Likes
1
Public
Repo makeup
Click a slice to open those files.
.json1.8 MB · 100%
At a glance
- Library
- tokenizers
- License
- cc-by-sa-3.0
- Access
- Public
- Created
- Jan 10, 2024
- Updated
- Jan 22, 2024
- SHA
- b0f9f169
Datasets
Model card
From the Hugging Face model README
WikiText-WordLevel
This is a simple word-level tokenizer created using the Tokenizers library. It was trained for educational purposes on the combined train, validation, and test splits of the WikiText-103 corpus.
- Tokenizer Type: Word-Level
- Vocabulary Size: 75K
- Special Tokens:
<s>(start of sequence),</s>(end of sequence),<unk>(unknown token) - Normalization: NFC (Normalization Form Canonical Composition), Strip, Lowercase
- Pre-tokenization: Whitespace
- Code: wikitext-wordlevel.py
The tokenizer can be used as simple as follows.
tokenizer = Tokenizer.from_pretrained('dustalov/wikitext-wordlevel')
tokenizer.encode("I'll see you soon").ids # => [68, 14, 2746, 577, 184, 595]
tokenizer.encode("I'll see you soon").tokens # => ['i', "'", 'll', 'see', 'you', 'soon']
tokenizer.decode([68, 14, 2746, 577, 184, 595]) # => "i ' ll see you soon"
- Library
- tokenizers
- License
- cc-by-sa-3.0
- Languages
- en
- Created
- Jan 10, 2024
- Updated
- Jan 22, 2024