Downloads · 30 days
0
aksara-tokenizer-20b
Ezekiel999/aksara-tokenizer-20b
aksara-tokenizer-20b is a machine learning model from Ezekiel999. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Byte-level BPE tokenizer for the AksaraLLM 20B pre-training run.
Access
Public
Updated Apr 24, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json9.5 MB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Apr 23, 2026
- Updated
- Apr 24, 2026
- SHA
- 01be583e
Model card
From the Hugging Face model README
AksaraLLM 20B Tokenizer
Byte-level BPE tokenizer for the AksaraLLM 20B pre-training run.
- Vocab size: 131,072
- Algorithm: Byte-level BPE (GPT-2 / LLaMA-3 style)
- Training corpus: ~12 GB balanced sample (English web / Indonesian web / Indonesian Wikipedia / Malay / Javanese / Sundanese) from FineWeb, FineWeb-2, CulturaX, and Wikipedia
- Produced by:
scripts/train_tokenizer_20b.pyin the AksaraLLM repo
Special tokens (pinned IDs)
The first 14 IDs are reserved for named special tokens, in this order:
| ID | Token |
|---|---|
| 0 | <|pad|> |
| 1 | <|bos|> |
| 2 | <|eos|> |
| 3 | <|unk|> |
| 4 | <|system|> |
| 5 | <|user|> |
| 6 | <|assistant|> |
| 7 | <|tool|> |
| 8 | <|im_start|> |
| 9 | <|im_end|> |
| 10 | <|fim_prefix|> |
| 11 | <|fim_middle|> |
| 12 | <|fim_suffix|> |
| 13 | <|endoftext|> |
The last 256 IDs (130816–131071) are reserved as <|reserved_N|> for future expansion without breaking already-pretrained checkpoints.
Fertility (tokens per whitespace-word)
Measured on ~200 KB held-out samples from each language:
| Language | Fertility | Target |
|---|---|---|
| English web | 1.280 | ≤ 1.40 |
| Indonesian wiki | 1.357 | ≤ 1.60 |
| Indonesian web (CulturaX) | 1.215 | ≤ 1.60 |
| Malay wiki | 1.368 | ≤ 1.60 |
| Javanese wiki | 1.657 | ≤ 1.80 |
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Ezekiel999/aksara-tokenizer-20b")
ids = tok("Halo dunia, saya AksaraLLM.", add_special_tokens=False).input_ids
# → 8 tokens
License
Apache-2.0.
- License
- apache-2.0
- Languages
- id, en, ms, jv, su
- Created
- Apr 23, 2026
- Updated
- Apr 24, 2026