Downloads · 30 days
0
tokenizertest
SupermAInds/tokenizertest
tokenizertest is a machine learning model from SupermAInds. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as unknown.
Access
Public
Updated Oct 27, 2024
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
Other1.5 KB · 75%
At a glance
- License
- unknown
- Access
- Public
- Created
- Oct 27, 2024
- Updated
- Oct 27, 2024
- SHA
- b8803199
Model card
From the Hugging Face model README
pip install transformers
from transformers import GPT2Tokenizer
Initialize the GPT-2 tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
Text to tokenize
text = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."
Tokenizing the text
tokens = tokenizer.tokenize(text) token_ids = tokenizer.convert_tokens_to_ids(tokens)
Print tokens and token IDs
print(tokens) print(token_ids)
- License
- unknown
- Created
- Oct 27, 2024
- Updated
- Oct 27, 2024