kl3m-multi-word-002-64k

KL3M Multi-Word Tokenizer v2 - 64K

This is the 65,536 token variant of the KL3M (Kelvin Legal Large Language Model) multi-word tokenizer family v2, optimized for legal domain text with hierarchical vocabulary nesting.

Overview

The KL3M multi-word tokenizers v2 are an improved family of byte-pair encoding (BPE) tokenizers trained on ~44GB of legal domain text from the KL3M dataset (copyright-clean legal corpus from the ALEA Institute). These tokenizers:

• Capture multi-word phrases as single tokens (e.g., "United States", "set forth", "accordance with")
• Encode complex legal terms efficiently (e.g., "Licensee", "hereinafter", "indemnification" as single tokens)
• Use hierarchical vocabulary nesting where smaller vocabularies are proper subsets of larger ones
• Outperform GPT-4 on legal text with 7.5% better compression on legal documents
• Enable vocabulary expansion experiments and transfer learning across vocabulary sizes

What's New in v2

• Cleaner special token design: 7 special tokens (removed experimental symbols)
• Improved legal domain optimization: Better encoding of common legal terminology
• Superior compression: 5.32 chars/token on legal text (vs 4.92 for GPT-4)
• Smaller file sizes: More efficient tokenizer representation

Multi-Word Tokenization Examples

What is multi-word tokenization? Unlike standard tokenizers that split text into subword pieces, these tokenizers capture multiple words as single tokens. This is especially powerful for legal text with common multi-word phrases.

Example: "The United States Department of Transportation is responsible for"

4K vocabulary (16 tokens):

[The][ ][United][ ][States][ ][Department][ of][ ][Trans][port][ation][ is][ ][responsible][ for]

32K vocabulary (10 tokens):

[The][ United][ States][ Department][ of][ ][Transportation][ is][ responsible][ for]

128K vocabulary (8 tokens) - "United States" is ONE token!

[The][ United States][ Department][ of][ Transportation][ is][ responsible][ for]

Example: "In accordance with the terms and conditions set forth herein"

4K vocabulary (16 tokens):

[In][ ][accordance][ with][ the][ ][terms][ and][ con][ditions][ ][set][ for][th][ ][herein]

32K vocabulary (10 tokens) - "accordance with" is ONE token!

[In][ ][accordance with][ the][ terms][ and][ conditions][ set][ forth][ herein]

128K vocabulary (8 tokens) - "accordance with" and "set forth" are single tokens!

[In][ accordance with][ the][ terms][ and][ conditions][ set forth][ herein]

Example: "The Supreme Court of the United States held that"

4K vocabulary (17 tokens):

[The][ S][up][re][me][ C][our][t][ of][ the][ ][United][ ][States][ h][eld][ that]

32K vocabulary (9 tokens):

[The][ Sup][reme][ Court][ of the][ United][ States][ held][ that]

128K vocabulary (7 tokens) - "United States" is ONE token!

[The][ Supreme][ Court][ of the][ United States][ held][ that]

Why This Matters

1. Shorter sequences = faster inference and training
2. Semantic coherence = "United States" as one unit, not two separate words
3. Better legal understanding = common legal phrases encoded atomically
4. Efficient compression = 7.5% fewer tokens than GPT-4 on legal text

Performance Comparison

On a realistic 3,743-character legal document (Software License Agreement):

Legal Terminology Efficiency

Common legal terms as single tokens (128K vocab):

Tokenizer Family

This tokenizer is part of a hierarchically nested family. Token IDs in smaller vocabularies are identical across all larger vocabularies, enabling seamless vocabulary expansion:

→ You are viewing: 65,536 (64K)

Key Features

1. Multi-Word Tokenization

Legal text contains frequent multi-word phrases that benefit from being treated as single tokens. The larger vocabularies capture increasingly sophisticated legal terminology:

4K vocabulary examples:

• Common legal particles: "herein", "thereof", "pursuant"
• Basic legal terms: "shall", "party", "agreement"

32K vocabulary examples:

• Complex terms: "Licensee", "Licensor", "jurisdiction", "arbitration"
• Multi-word phrases: "intellectual property", "force majeure"

128K vocabulary examples:

• Specialized terms: "indemnification", "confidentiality", "non-exclusive"
• Complete phrases: "representations and warranties", "WHEREAS"

2. Hierarchical Vocabulary Nesting

Token IDs 0-4,095 are identical across all tokenizer sizes. This enables:

• Vocabulary expansion during training: Start with 4K vocab, expand to 32K mid-training
• Transfer learning: Initialize larger vocab models from smaller vocab checkpoints
• Controlled ablations: Compare vocab sizes while maintaining token alignment
• Model compression: Train with large vocab, deploy with smaller vocab

3. Legal Domain Optimization

Trained on the KL3M corpus (44GB of legal text):

• Court opinions and case law
• Contracts and agreements
• Patents and IP documents
• Legal briefs and filings
• Statutory and regulatory text

This specialized training produces:

• Better compression on legal documents (5.32 chars/token vs 4.92 for GPT-4)
• Semantic coherence for legal multi-word expressions
• Reduced sequence lengths leading to faster inference

4. Special Tokens (v2)

Seven essential special tokens for language model training:

Note: v2 removes experimental symbols (⧈, ⚖, ⏵) from v1 for cleaner design.

Usage

With Transformers

from transformers import PreTrainedTokenizerFast

# Load tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained(
    "alea-institute/kl3m-multi-word-002-64k"
)

# Tokenize text
text = "The Licensor hereby grants to Licensee a non-exclusive license."
tokens = tokenizer.encode(text)
print(f"Tokens: {len(tokens)}")

# Decode
decoded = tokenizer.decode(tokens)
print(f"Decoded: {decoded}")

With tokenizers Library

from tokenizers import Tokenizer

# Load tokenizer
tokenizer = Tokenizer.from_pretrained(
    "alea-institute/kl3m-multi-word-002-64k"
)

# Encode
encoding = tokenizer.encode(text)
print(f"Tokens: {encoding.tokens}")
print(f"IDs: {encoding.ids}")

Training a Model

from transformers import AutoConfig, AutoModelForMaskedLM, PreTrainedTokenizerFast

# Load tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained(
    "alea-institute/kl3m-multi-word-002-64k"
)

# Create model config
config = AutoConfig.from_pretrained(
    "bert-base-uncased",
    vocab_size=tokenizer.vocab_size,
)

# Initialize model
model = AutoModelForMaskedLM.from_config(config)

# Train with HuggingFace Trainer...

Technical Details

Training Corpus

• Source: KL3M (Kelvin Legal Large Language Model) dataset
• Size: ~44.2 GB (44,168,540,153 bytes)
• Lines: 1,018,355,750
• Words: 5,997,814,602
• Domain: Legal documents (copyright-clean)

Training Parameters

bbpe train \
  --max-entropy 7.0 \
  --preprocessor unicode-whitespace \
  --preprocessor-probability 0.1 \
  --vocab-size 131072 \
  --family-size 65536 --family-size 32768 \
  --family-size 16384 --family-size 8192 --family-size 4096

• Max entropy: 7.0 (balances multi-word phrases with common tokens)
• Preprocessing: Unicode whitespace normalization (10% probability)
• Byte fallback: Enabled (handles any input)

Vocabulary Structure

• Base vocabulary: 256 bytes + 49 extended chars = 305 base tokens
• Learned merges: vocab_size - 305 - 7 (special tokens)
• Nesting property: All tokens in size N exist in size 2N

Recommendations

By Use Case

Legal Document Processing (contracts, patents, briefs):

• Best: 128K or 64K vocab
• Rationale: Maximum compression of legal terminology
• Benefit: Shorter sequences, faster inference

Resource-Constrained Environments:

• Best: 16K or 32K vocab
• Rationale: Good balance of compression and model size
• Benefit: Smaller embedding layers, less memory

Experimentation / Research:

• Best: Multiple sizes with vocabulary expansion
• Rationale: Leverage nested structure for novel training strategies
• Benefit: Test curriculum learning, transfer learning

Model Size Guidelines

Choose vocab size based on your model parameter count:

Limitations

• Training domain: Optimized for legal English text; may underperform on other domains
• Multilingual: Trained primarily on English; limited non-English support
• Code: Less optimized for code compared to code-specific tokenizers
• Vocabulary size: Larger vocabs (64K+) require more embedding memory

Citation

If you use these tokenizers in your research, please cite:

@misc{kl3m-multi-word-002,
  title={KL3M Multi-Word Tokenizers v2: Hierarchically Nested BPE for Legal Domain},
  author={ALEA Institute},
  year={2025},
  publisher={HuggingFace},
  url={https://huggingface.co/alea-institute/kl3m-multi-word-002-64k}
}

License

MIT License

About ALEA Institute

The ALEA Institute develops open-source tools and datasets for legal AI, including the KL3M corpus and multi-word tokenizers.

Related Resources

• KL3M Dataset: aleainstitute.ai/work/kl3m
• bbpe Tokenizer Trainer: github.com/microsoft/blingfire
• v1 Tokenizers: alea-institute/kl3m-multi-word-001-*

Version History

• v2 (002 series): Improved special token design, better legal optimization
• v1 (001 series): Initial release with 10 special tokens

Contact

For questions or issues, please contact: [contact information TBD]