Downloads · 30 days
0
arka-v2-tokenizer-64k
Abhayn01/arka-v2-tokenizer-64k
arka-v2-tokenizer-64k is a machine learning model from Abhayn01. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Custom 64K Unigram tokenizer trained for the Arka v2 model, optimized for reasoning, math, and code.
Access
Public
Updated Jul 12, 2026
Repo size
10.2 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json4.5 MB · 77%
At a glance
- Access
- Public
- Created
- Jul 11, 2026
- Updated
- Jul 12, 2026
- SHA
- 8dae5fb4
Model card
From the Hugging Face model README
Arka v2 Tokenizer (64K) - Fixed Fast Tokenizer
Custom 64K Unigram tokenizer trained for the Arka v2 model, optimized for reasoning, math, and code.
Update: This version correctly implements the Metaspace pre-tokenizer using SentencePieceUnigramTokenizer.from_spm(). Spaces are properly handled, and the token bug has been resolved (V2).
Configuration
- Vocab Size: 64000
- Context Length: 8192
- Algorithm: SentencePiece Unigram + Byte-fallback
- Clean-up Spaces: False (Preserves code formatting)
- Data Ratio: 60% Edu-Web, 20% Code, 15% Math, 5% StackExchange QA
- Created
- Jul 11, 2026
- Updated
- Jul 12, 2026