medical-bpe-16k

medical-bpe-16k

A 16,000-token byte-level BPE tokenizer trained on 45,000 PubMed abstracts.

On biomedical text it encodes more efficiently than cl100k_base (GPT-4) and o200k_base (GPT-4o), despite a vocabulary 6–12x smaller. On general English it is clearly worse than both. Both halves of that sentence are the point — a domain tokenizer reallocates vocabulary, it does not add any.

Results

Fertility is tokens per whitespace word: lower is better. Evaluated on 5,000 held-out PubMed abstracts and 5,000 held-out wikitext-103 paragraphs, neither of which appeared in training.

Biomedical text (held-out PubMed, n=5,000)

5.8% fewer tokens per abstract than GPT-4's tokenizer, 3.8% fewer than GPT-4o's.

General English (held-out wikitext-103, n=5,000)

35.6% more tokens than o200k on ordinary prose. This tokenizer is not a general-purpose replacement.

Single-token coverage

Out of 20 common clinical terms, encoded as exactly one token:

Why it wins

acetylcholinesterase inhibitor costs 3 pieces here, 6 under o200k, 7 under cl100k, and 10 under a same-sized BPE trained on wikitext.

Trained on abstracts, the merge table spends its early slots learning that -ation, -osis, -emia and -inib are load-bearing morphology. Trained on Wikipedia, the same budget goes to Ġ, and Ġ. — the first merges of the control tokenizer are punctuation, while this one's are in, ti, al, ra.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("salmanhadli/medical-bpe-16k")

tok.tokenize("acetylcholinesterase inhibitor")
# ['acetyl', 'cholinesterase', 'Ġinhibitor']

Or with the tokenizers library directly:

from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download

tok = Tokenizer.from_file(
    hf_hub_download("salmanhadli/medical-bpe-16k", "tokenizer.json")
)
tok.encode("serum creatinine 1.4 mg/dL").tokens

Training details

There is no unk_token — byte-level BPE has full byte coverage, so any UTF-8 input round-trips losslessly.

Intended use and limitations

Intended for tokenizing biomedical text: abstracts, clinical notes, literature corpora. Useful when context length or token cost is the binding constraint on a domain pipeline.

Not intended for general-purpose text, where it is measurably worse than the standard tokenizers above. It is a tokenizer only — there are no model weights here, and it carries no medical knowledge and makes no clinical claims. Any model trained on it inherits whatever biases live in PubMed abstracts: a literature corpus is not a patient population.

The vocabulary reflects abstract prose. Clinical shorthand, drug names post-dating the corpus, and non-English text will fragment more than the numbers above suggest.

Reproducing

Trained with tokenization-explainer:

uv run python scripts/train_medical_tokenizer.py \
  --corpus data/pubmed_train.jsonl \
  --vocab-size 16000 \
  --out artifacts/medical-bpe-pubmed/tokenizer.json

The comparison above is scripts/compare_tokenizers.py, which also trains the same-sized wikitext control so the win can be attributed to the corpus rather than to the algorithm or the vocabulary size.

License

MIT