blackwood-0.7b-bpe49152-tokenizer

Byte-Level BPE Tokenizer (vocab = 49,152)

A production-grade byte-level BPE tokenizer trained on a mixture of permissively-licensed English text, code (via FIM) and math, with 256 reserved special-token slots covering ChatML / chain-of-thought / tool-call / document-boundary primitives.

The companion pretokenized corpus that uses this exact tokenizer.json is available at BlackwoodAI/blackwood-pretrain-49152.

At a glance

The 256 special-token block is large by design — the first ~26 IDs cover common roles (chat, CoT, tool-call, document/source boundaries), and the remaining 230 slots are reserved for downstream fine-tune specialisations so new roles can be aliased without retraining BPE merges.

Special-token catalog (IDs 0–255)

256 reserved slots, stable across releases.

Core (IDs 0–2)

Chat / ChatML (IDs 3–7)

Reasoning / CoT (IDs 8–11)

Tool calling — generic (IDs 12–13)

Document structure (IDs 14–15)

Extended markers (IDs 16–25)

For long-horizon tool / planning workflows. Compose with the generic tool-call wrappers.

Reserved (IDs 26–255)

<\|reserved_010\|> through <\|reserved_239\|> — 230 slots reserved for downstream fine-tune specialisations.

Measured fertility (tok/char, real-corpus samples)

Numbers reproduced from test_report.json:

Notes:

• English prose at 0.22 is competitive with Llama-3-class tokenizers.
• Non-Latin scripts (CJK, Arabic) are not optimised — this is an English / code / math tokenizer.
• Any UTF-8 input is covered losslessly thanks to ByteLevel.

Training corpus

Trained on 15 GB of UTF-8 text streamed across 7 sources (see stats.json for exact byte / doc counts):

All sources hold permissive licenses (ODC-BY / CC-BY-4.0 / various permissive).

Round-trip + correctness tests

Released with a 6-test verification suite:

1. Vocab size == 49,152
2. All 256 special tokens at correct sequential IDs (0–255)
3. Special tokens encode as single atomic tokens
4. Round-trip decode(encode(x)) == x exact on synthetic samples
5. Round-trip exact on real corpus samples across every source
6. ChatML / CoT / tool-call template strings parse as expected token streams

All six pass; details in test_report.json.

Usage

Raw tokenizers library

from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download

tok_path = hf_hub_download(
    repo_id="BlackwoodAI/blackwood-0.7b-bpe49152-tokenizer",
    filename="tokenizer.json",
)
tok = Tokenizer.from_file(tok_path)
ids = tok.encode("<|im_start|><|user|>hi<|im_end|>").ids
print([tok.id_to_token(i) for i in ids])
# ['<|im_start|>', '<|user|>', 'hi', '<|im_end|>']

transformers

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(
    "BlackwoodAI/blackwood-0.7b-bpe49152-tokenizer",
    use_fast=True,
)

ChatML inference prompt

prompt = (
    "<|im_start|><|system|>You are a helpful coding assistant.<|im_end|>\n"
    "<|im_start|><|user|>Write fizzbuzz in Python.<|im_end|>\n"
    "<|im_start|><|assistant|>"
)
input_ids = tok.encode(prompt).ids

CoT template

prompt = (
    "<|im_start|><|user|>What's 17 × 23?<|im_end|>\n"
    "<|im_start|><|assistant|>"
    "<think>17 × 20 = 340, 17 × 3 = 51, total = 391.</think>"
    "<answer>391</answer><|im_end|>"
)

Tool-call template

prompt = (
    "<|im_start|><|user|>What's the weather in NYC?<|im_end|>\n"
    "<|im_start|><|assistant|>"
    "<|tool_call|><|fn_name|>get_weather"
    "<|fn_args|>{\"city\":\"NYC\"}"
    "<|tool_result|><|fn_response|>{\"temp_f\":72,\"condition\":\"clear\"}"
    "<answer>It's 72°F and clear in NYC.</answer>"
    "<|terminate|><|im_end|>"
)

Notes on byte-level encoding

• add_prefix_space=False: encoding "hello" differs from " hello" by the first token. Be consistent at training and inference.
• No normaliser is applied. Casing, accents, control bytes and Unicode variants are all preserved. Layer NFKC normalisation before encoding if your downstream needs it.
• No subword regularisation (BPE-Dropout, SentencePiece unigram sampling) is applied at the tokenizer level. Add it downstream if your use case needs it.

License

Tokenizer artifact: Apache-2.0. Training corpora retain their upstream licenses (ODC-BY / CC-BY-4.0 / various).