Downloads · 30 days
0
wikitext2-bpe-30k
PhilixOkigbo/wikitext2-bpe-30k
wikitext2-bpe-30k is a machine learning model from PhilixOkigbo. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
A byte-level Byte-Pair Encoding tokenizer trained from scratch on the WikiText-2 (wikitext-2-v1) corpus.
Access
Public
Updated Aug 31, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json2.1 MB · 100%
At a glance
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Aug 31, 2026
- Updated
- Aug 31, 2026
- SHA
- 106d4d6b
Datasets
Model card
From the Hugging Face model README
wikitext2-bpe-30k
A byte-level Byte-Pair Encoding tokenizer trained from scratch on the WikiText-2
(wikitext-2-v1) corpus.
Configuration
| Setting | Value |
|---|---|
| Algorithm | Byte-level BPE |
| Target vocabulary | 30,000 |
| Realised vocabulary | 30,000 |
| Minimum pair frequency | 2 |
| Normalizer | NFKC (no lowercasing) |
| Pre-tokenizer | ByteLevel (add_prefix_space=False) |
| Post-processor | [CLS] $A [SEP] |
| Special tokens | [PAD], [UNK], [CLS], [SEP], [MASK] |
| Training rows | 21,329 (cleaned and exactly deduplicated) |
Preprocessing
wikitext-2-v1 is the word-level WikiText subset, so the training text was cleaned first:
<unk> placeholders removed, @-@ / @.@ / @,@ artifacts reversed, detached punctuation and
possessives re-attached, whitespace collapsed. Exact deduplication was applied to the training
split only.
Evaluation
| validation | test | |
|---|---|---|
| lines evaluated | 2461 | 2858 |
| total tokens | 209667 | 233147 |
| round-trip exact (%) | 98.618 | 99.895 |
| deterministic (%) | 100 | 100 |
| [UNK] rate (%) | 0 | 0 |
| avg tokens/line | 85.2 | 81.58 |
| median tokens/line | 72 | 66 |
| p95 tokens/line | 227 | 229 |
| chars per token | 4.807 | 4.782 |
| tokens per word | 1.197 | 1.197 |
Because the base alphabet contains all 256 bytes, [UNK] is never emitted and decoding is lossless.
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("PhilixOkigbo/wikitext2-bpe-30k")
print(tok("Byte-pair encoding is a subword tokenization algorithm."))
Trained by Chinedu Okigbo.
- Library
- transformers
- License
- mit
- Languages
- en
- Created
- Aug 31, 2026
- Updated
- Aug 31, 2026