Downloads · 30 days
0
gpt-neox-tokenizer-digits
NeelNanda/gpt-neox-tokenizer-digits
gpt-neox-tokenizer-digits is a machine learning model from NeelNanda. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This is a fork of the GPT NeoX 20B tokenizer, edited to split every numerical digit into a separate token. This has the goal of making it easier for the model to learn arithmetic capabilities and to hopefully be more…
Access
Public
Updated Nov 28, 2023
Repo size
—
Likes
2
Public
Repo makeup
Click a slice to open those files.
.json2 MB · 100%
At a glance
- Access
- Public
- Created
- Oct 12, 2022
- Updated
- Nov 28, 2023
- SHA
- 0f667157
Model card
From the Hugging Face model README
This is a fork of the GPT NeoX 20B tokenizer, edited to split every numerical digit into a separate token. This has the goal of making it easier for the model to learn arithmetic capabilities and to hopefully be more interpretable, and copies the idea from the PaLM tokenizer.
This was done, extremely hackily, by just removing every token that contained "\d\d" (eg "2013"). All remaining digit containing tokens are "0" ... "9" and " 0" ... " 9"
This comes at the cost of making modelling normal text harder, since eg dates like 2013 which naturally should be a single token are now 2|0|1|3.
This has a reduced vocab size of 48252 (several of the tokens towards the end are special whitespace tokens copied in from GPT-NeoX to make tokenizing code easier - some of these are duplicated in the vocabulary and thus may not actually show up at train time).
It includes a padding token (<|PAD|>) an End-Of-String token (<|EOS|>) and a Beginning-Of-String token (<|BOS|>)
- Created
- Oct 12, 2022
- Updated
- Nov 28, 2023