medical-bpe-16k

medical-bpe-16k

A domain-specific byte-level BPE tokenizer trained on biomedical text, built to test one specific claim: for specialized text, what the tokenizer was trained on matters more than how big its vocabulary is.

What this is

• Algorithm: Byte-level Byte-Pair Encoding (BPE)
• Vocabulary size: 16,000 tokens
• Training corpus: 45,000 PubMed abstracts, streamed from the Hugging Face Hub
• Held-out evaluation set: 5,000 PubMed abstracts, excluded from training

Why it exists

A general-purpose tokenizer (e.g., GPT-4's cl100k, ~100K vocab) fragments specialized biomedical vocabulary — drug names, biochemical terms, diagnostic codes — into multiple pieces, even with a vocabulary 6x larger than this one. This tokenizer is a controlled test of that claim: trained on domain text at a small (16K) vocabulary, and compared against a same-size tokenizer trained on general text (general-bpe, on WikiText-103) to isolate whether domain or vocabulary size actually drives tokenization efficiency.

Evaluation (fertility — average tokens per word, lower is better)

Held-out PubMed abstracts (5,000 docs, never seen during training):

Held-out WikiText-103 (5,000 docs, general English) — the flip side:

The key comparison is custom-med vs. general-bpe — identical vocabulary size (16,000), identical algorithm, trained on different corpora. custom-med wins on medical text (1.375 vs. 1.747) and loses on general text (1.577 vs. 1.208) — a clean crossover that isolates domain training, not vocabulary size, as the cause. This tokenizer even outperforms general-purpose tokenizers with vocabularies up to 12x larger (o200k, cl100k) on medical text specifically.

Usage

from transformers import PreTrainedTokenizerFast

tok = PreTrainedTokenizerFast.from_pretrained("prasadbhoite/medical-bpe-16k")
print(tok.tokenize("The patient was prescribed empagliflozin for management of SGLT2-related complications."))

Limitations

• Trained only on PubMed abstracts (formal, published biomedical writing) — not evaluated on informal clinical notes, which use different shorthand, abbreviations, and structure (e.g., ICU/discharge notes).
• Its efficiency gain is domain-specific by design: general-English fertility is worse than any general-purpose tokenizer tested, since its entire 16K-token budget was spent on medical morphemes.
• This is a tokenizer only — it has not been paired with a language model trained to use it. Its token IDs are not compatible with any existing pretrained model's embedding table without further adaptation (vocabulary extension + continued pre-training).

Context

Built as a hands-on exercise to demonstrate BPE mechanics, domain-specific tokenizer training, and fertility-based evaluation methodology.