Downloads · 30 days
0
malayalam-llama-2-tokenizer-v0.1
learninbit/malayalam-llama-2-tokenizer-v0.1
malayalam-llama-2-tokenizer-v0.1 is a text classification model from learninbit. Use it when you need a label for a piece of text. It is set up for transformers. The card lists the license as mit.
- This tokenizer was trained using the CulturaX dataset. We sample 1.2 million datapoints at random. - This was trained using the SentencePiece by Google. - Then the trained tokens were then added to the LlamaTokenize…
Access
Public
Updated Jan 26, 2024
Repo size
906 KB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.model906 KB · 100%
At a glance
- Task
- Text Classification
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Jan 26, 2024
- Updated
- Jan 26, 2024
- SHA
- 44702659
Datasets
Model card
From the Hugging Face model README
About
- This tokenizer was trained using the CulturaX dataset. We sample 1.2 million datapoints at random.
- This was trained using the SentencePiece by Google.
- Then the trained tokens were then added to the
LlamaTokenizerleading to a total of 49,120 tokens from 32,000 from the original tokenizer. - The merging was done according to what the Chinese-Llama-Alpaca's merging did.
Usage
from transformers import LlamaTokenizer
tokenizer = LlamaTokenizer.from_pretrained("learninbit/malayalam-llama-2-tokenizer-v0.1")
text = "ഹനഫസ ഹഫഞ്ചഥ ചകഡു ടെണല ഡൃൊമത്തീഴ ടഞ്ഞഭഞ റദ്ധഷ ഌിപത്മഫഥ ടജ്ജഡ്ഡപ്പെവ പഴുണൊ."
tokens = tokenizer.tokenizer(text)
- Task
- Text Classification
- Library
- transformers
- License
- mit
- Languages
- ml
- Created
- Jan 26, 2024
- Updated
- Jan 26, 2024