Downloads · 30 days
0
wikitext-103-tokenizer
Kristijan/wikitext-103-tokenizer
wikitext-103-tokenizer is a machine learning model from Kristijan. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This is a BPE tokenizer retrained from scratch on the concatenated Wikitext-103 train, evaluation, and test sets. The vocabulary had 28,439 entries.
Access
Public
Updated Oct 26, 2022
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json454 KB · 63%
At a glance
- Access
- Public
- Created
- Oct 26, 2022
- Updated
- Oct 26, 2022
- SHA
- 347b9036
Model card
From the Hugging Face model README
Model info
This is a BPE tokenizer retrained from scratch on the concatenated Wikitext-103 train, evaluation, and test sets. The vocabulary had 28,439 entries.
This tokenizer was use to tokenize text for the GPT-2 model trained on Wikitext-103.
Usage
You can download the tokenizer directly from hub as follows:
from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained("Kristijan/wikitext-103-tokenizer")
After cloning/downloading the files, you can load the tokenizer using the /from_pretrained() methods as follows:
from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained(path_to_folder_with_merges_and_vocab_files)
- Created
- Oct 26, 2022
- Updated
- Oct 26, 2022