Downloads · 30 days
0
arlowgpt-tokenizer-v2
yuchenxie/arlowgpt-tokenizer-v2
arlowgpt-tokenizer-v2 is a machine learning model from yuchenxie. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
This repository contains a custom-trained BPE tokenizer for ArlowGPT, created by Yuchen Xie.
Access
Public
Updated Feb 1, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json10.7 MB · 91%
At a glance
- Library
- transformers
- License
- apache-2.0
- Access
- Public
- Created
- Feb 1, 2025
- Updated
- Feb 1, 2025
- SHA
- fc0e3e6f
Base models
Datasets
Model card
From the Hugging Face model README
ArlowGPT Tokenizer
This repository contains a custom-trained BPE tokenizer for ArlowGPT, created by Yuchen Xie.
Tokenizer Details
- Type: BPE (Byte-Pair Encoding)
- Vocabulary Size: 131,072 tokens
- Special Tokens:
- Start of Text: <|startoftext|>
- End of Text: <|endoftext|>
- Padding: <|pad|>
- Unknown: <|unk|>
- Mask: <|mask|>
- Message Start: <|im_start|>
- Message End: <|im_end|>
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("yuchenxie/arlowgpt-tokenizer-v2")
Training Details
This tokenizer was trained on the 10B GPT-2 randomly shuffled tokens under a custom script composed by Yuchen Xie. This tokenizer is compatible with HuggingFace Transformer's Auto Tokenizer class.
- Library
- transformers
- License
- apache-2.0
- Languages
- en
- Created
- Feb 1, 2025
- Updated
- Feb 1, 2025