Downloads · 30 days
0
wikitext-103_tokenizer_v2
Kristijan/wikitext-103_tokenizer_v2
wikitext-103_tokenizer_v2 is a machine learning model from Kristijan. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This is a BPE tokenizer retrained from scratch on the Wikitext-103 train, evaluation, and test sets. The vocabulary had 28,439 entries.
Access
Public
Updated Mar 30, 2023
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json454 KB · 63%
At a glance
- Access
- Public
- Created
- Mar 30, 2023
- Updated
- Mar 30, 2023
- SHA
- ec05ae64
Model card
From the Hugging Face model README
Model info
This is a BPE tokenizer retrained from scratch on the Wikitext-103 train, evaluation, and test sets. The vocabulary had 28,439 entries.
This tokenizer was used to tokenize text for the GPT-2-like transformer language model trained on Wikitext-103.
Usage
You can download the tokenizer directly from hub as follows:
from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained("Kristijan/wikitext-103-tokenizer_v2")
After cloning/downloading the files, you can load the tokenizer using the /from_pretrained() methods as follows:
from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained(path_to_folder_with_merges_and_vocab_files)
- Created
- Mar 30, 2023
- Updated
- Mar 30, 2023