normster

Llama-2-tokenizer

normster/Llama-2-tokenizer

Llama-2-tokenizer is a machine learning model from normster. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama2.

This repo contains the tokenizer for the Llama-2 models. The tokenizer is under a non-commercial license (see the LICENSE file). You should only use this repository if you have been granted access to the model by fill…