NexTok-32K

NexTok-32K

NexTok-32K is a custom 32,000-vocabulary byte-level BPE tokenizer developed for a from-scratch language model focused on computer science, programming, mathematics, educational text, and general natural language.

Overview

The tokenizer uses a 32K vocabulary as a deliberate trade-off between tokenization efficiency and vocabulary-dependent model parameter cost.

Training Data

Approximately 1,000,000 samples were used with the following target mixture:

The mixture was selected to provide coverage of general educational prose, mathematics, programming, technical discussions, and code.

Tokenizer Configuration

The tokenizer uses the following GPT-4-style regex:

'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}++|\p{N}{1,3}+| ?[^\s\p{L}\p{N}]++[\r\n]*+|\s++$|\s*[\r\n]|\s+(?!\S)|\s

ByteLevel settings:

add_prefix_space = false
use_regex = false

No normalizer is used.

Held-Out Benchmark

NexTok-32K was compared with the Cosmo2 tokenizer (49,152 vocabulary) on held-out benchmark datasets. The evaluation measures tokenizer efficiency, not model accuracy.

Ratio interpretation: values below 1.0 mean NexTok-32K used fewer tokens.

Result

On this evaluation, NexTok-32K used 0.64% more tokens overall than Cosmo2.

NexTok-32K used fewer tokens on:

• MMLU: 0.84% fewer
• GSM8K: 1.57% fewer

Cosmo2 used fewer tokens on:

• ARC-Challenge: 1.06% fewer
• HellaSwag: 2.48% fewer
• Winogrande: 4.39% fewer

Both tokenizers had 0 round-trip failures across all 13,758 evaluated samples.

This is a tokenizer-efficiency comparison only. It does not establish downstream model performance. The vocabulary sizes also differ: 32K vs 49,152.

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "Abhay2310/NexTok-32K"
)

text = "Explain binary search."

ids = tokenizer.encode(
    text,
    add_special_tokens=False
)

decoded = tokenizer.decode(
    ids,
    skip_special_tokens=False
)

print("Tokens:", tokenizer.tokenize(text))
print("Token IDs:", ids)
print("Token count:", len(ids))
print("Decoded:", decoded)
print("Round-trip:", decoded == text)

Special Tokens

The chat tokens are reserved for future instruction or conversational training; they were not evidence that chat-formatted data was used during tokenizer training.

Limitations

• The 32K vocabulary is smaller than Cosmo2's 49,152 vocabulary.
• Training data is strongly oriented toward English educational, mathematical, technical, and programming text.
• The code component was Python-focused; other programming languages have not yet been benchmarked.
• The current held-out evaluation is preliminary and measures tokenization efficiency rather than downstream language-model quality.

Status

Experimental — Active Development

NexTok-32K is the tokenizer component of an ongoing from-scratch language-model project.

Citation

@misc{kashyap2026nextok32k,
  title        = {NexTok-32K},
  author       = {Kashyap, Abhay J.},
  year         = {2026},
  howpublished = {Hugging Face Hub}
}