Downloads · 30 days
0
small-german-tokenizer
qikp/small-german-tokenizer
small-german-tokenizer is a machine learning model from qikp. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as cc0-1.0.
This is a small public domain-like tokenizer optimized for German.
Access
Public
Updated Mar 7, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json542 KB · 100%
At a glance
- Library
- transformers
- License
- cc0-1.0
- Access
- Public
- Created
- Mar 7, 2026
- Updated
- Mar 7, 2026
- SHA
- 7f6d5fe4
Datasets
Spaces
Model card
From the Hugging Face model README
Small German Tokenizer
This is a small public domain-like tokenizer optimized for German.
Special Tokens
- End-of-Sequence token:
[EOS] - Padding token:
[PAD]
Training
This tokenizer was trained on the context column of the configs task1 and task4 in deutsche-telekom/Ger-RAG-eval.
Limitations
Due to its small corpus, this tokenizer may split words into smaller pieces. Also, some uncommon special tokens aren't present, you'll have to add them manually if needed.
- Library
- transformers
- License
- cc0-1.0
- Languages
- de
- Created
- Mar 7, 2026
- Updated
- Mar 7, 2026