Supernova-Nepali-Tokenizer-V3

Supernova Nepali Tokenizer V3 (Absolute Power)

Supernova V3 is a scratch-built tokenizer infrastructure that achieves 7x the efficiency and speed of generic tokenizers (like Tiktoken or HuggingFace Fast Tokenizers) for the Nepali language.

Why to use this model?

• Speed: 1.37x+ faster than older one on standard CPUs.
• Power: 7x more efficient character-to-token representation for Devanagari.then older
• No GPU Needed: Designed for high-speed inference on edge devices and standard

How to use

# Load the tokenizer_v3.json and use the encode/decode logic provided in the Supernova class.

Status: Shipped. The era of GPU-dependency for Nepali NLP ends here. Supernova Nepali Tokenizer V3

A lightweight, CPU-oriented Nepali tokenizer using a custom Trie-based Max-Match Priority Encoder.

Installation

pip install huggingface_hub

Download

from huggingface_hub import hf_hub_download

tokenizer_path = hf_hub_download( repo_id="Supernova11c/Supernova-Nepali-Tokenizer-V3", filename="tokenizer.json" )

print("Tokenizer:", tokenizer_path)

Important

This repository contains a custom tokenizer format.

It is not a standard "AutoTokenizer" repository.

The "tokenizer.json" file contains the tokenizer vocabulary, merge definitions, and architecture metadata.

Architecture

• Architecture: Trie-based Max-Match Priority Encoder
• Version: 3.0.0
• Vocabulary size: 348
• Merge count: 92
• Designed for CPU inference
• Focus: Nepali and mixed Nepali/English text

Loading

import json

with open(tokenizer_path, "r", encoding="utf-8") as f: config = json.load(f)

print(config["model_name"]) print(config["version"]) print(config["architecture"]) print("Vocabulary:", len(config["vocab"])) print("Merges:", len(config["merges"]))

Benchmarking

For reproducible comparisons, benchmark all tokenizers on the same corpus and CPU.

Recommended metrics:

• Mean latency
• Median latency
• Best latency
• Characters/second
• Bytes/second
• Tokens/second
• Tokens/character
• Round-trip reconstruction accuracy
• Memory usage

Do not compare token counts alone because different tokenizers can represent the same text using very different numbers of tokens.

Benchmark claim

Benchmark results should always include:

• hardware/CPU information
• dataset size
• dataset composition
• number of repetitions
• warm-up procedure
• tokenizer versions
• exact benchmark code

Performance claims should be reported from measured results rather than assumed architectural advantages.