qikp

bpe-character-tokenizer

qikp/bpe-character-tokenizer

bpe-character-tokenizer is a machine learning model from qikp. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc0-1.0.

Simple pika- / GPT-2- like BPE tokenizer that was trained on every possible byte. The script used to generate this was included.