manangarg

ind-llm-tokenizer

manangarg/ind-llm-tokenizer

ind-llm-tokenizer is a machine learning model from manangarg. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.

• Indic Language LLM Tokenizer - This is an indic language NLP tokenizer which is merged with LLaMA 2 tokenizer.