dutch-llama-tokenizer

Dutch-Llama Tokenizer

Overview

The Dutch-Llama Tokenizer is a versatile tokenizer trained to handle a variety of languages and formats, including Dutch, English, Python code, Markdown, and general text. It's based on a dataset consisting of diverse sources, which ensures its capability to tokenize a wide range of text inputs effectively.

Dataset Composition

The tokenizer was trained on a comprehensive dataset, including:

• MC4 Dutch and English texts (195M)
• English and Dutch Wikipedia (278M and 356M, respectively)
• Dutch and English book datasets (211M and 355M, respectively)
• Dutch news articles (256M)
• CodeParrot GitHub Python code (158M)
• CodeSearchNet Python code (126M)
• Markdown files with math markup (5.8M)
• Arxiv scientific papers (169M)

Tokenizer Settings

The tokenizer was trained using the spm_train command with the following settings:

• Model Type: Byte Pair Encoding (BPE)
• Vocab Size: 32,000
• Character Coverage: 100%
• Support for splitting digits and whitespace-only pieces
• Optimized for large corpus training
• Byte Fallback and language acceptance for Dutch (nl) and English (en)
• Special tokens and IDs for unknown, beginning of sentence, end of sentence, padding, and custom user-defined symbols

Installation

To use the Dutch-Llama Tokenizer, ensure you have Python 3.10.12 or later installed. Then, install the Transformers library from Hugging Face:

pip install transformers

Usage

First, import the AutoTokenizer from the Transformers library and load the Dutch-Llama Tokenizer:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("yhavinga/dutch-llama-tokenizer")

To tokenize text, use the tokenizer.tokenize method. For converting tokens to IDs and decoding them back to text, use tokenizer.convert_tokens_to_ids and tokenizer.decode respectively:

# Example text
text = "Steenvliegen of oevervliegen[2] (Plecoptera) 华为发布Mate60手机"

# Tokenization and decoding
tokens = tokenizer.tokenize(text)
token_ids = tokenizer.convert_tokens_to_ids(tokens)
decoded_text = tokenizer.decode(token_ids)

print(decoded_text)

Dutch Tokenizer Arena

Compare the effectiveness of this tokenizer on different inputs at the Hugging Face Space: Dutch Tokenizer Arena.

Comparison with Other Tokenizers

The following table shows the number of tokens produced by the Dutch-Llama Tokenizer, the Mistral Tokenizer, the GroNLP GPT-2 Dutch Tokenizer, and the UL2 Dutch Tokenizer on a variety of inputs.

🇳🇱 🇧🇪🐍📐