solar-pro4-tokenizer

Solar Pro 4 Tokenizer

Tokenizer for Upstage Solar Pro 4, with a vocabulary of 196,608 tokens and support for English, Korean, Japanese, and more.

This repository contains tokenizer files for encoding text and estimating text token counts. It does not contain model weights or a chat template.

Usage

Install the Transformers library:

pip install transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("upstage/solar-pro4-tokenizer")

text = "Hi, how are you?"
token_ids = tokenizer.encode(text, add_special_tokens=False)

print("Token IDs:", token_ids)
print("Tokens:", tokenizer.convert_ids_to_tokens(token_ids))
print("Number of tokens:", len(token_ids))
print("Decoded text:", tokenizer.decode(token_ids, skip_special_tokens=False))

To load a downloaded copy without network access:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "./solar-pro4-tokenizer",
    local_files_only=True,
    trust_remote_code=False,
)

These examples count tokens in the supplied text. A full API request can also include message formatting, system instructions, and tool definitions. Use the API response's usage fields for the actual request token usage.

See the token counting guide for more information.

License

See Upstage Solar License.