rasyosef

gpt2-oscar-amharic-tokenizer

rasyosef/gpt2-oscar-amharic-tokenizer

gpt2-oscar-amharic-tokenizer is a machine learning model from rasyosef. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.

This repo contains a Byte-Pair Encoding tokenizer trained on the Amharic subset of the oscar dataset. It's the same as the GPT-2 tokenizer but trained from scratch on an amharic dataset with a vocabulary size of 24000.