Downloads · 30 days
0
xlm-roberta-tokenizer-50k
metythorn/xlm-roberta-tokenizer-50k
xlm-roberta-tokenizer-50k is a machine learning model from metythorn. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This is a custom tokenizer trained for Khmer language based on XLM-RoBERTa architecture.
Access
Public
Updated Jul 10, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json3.9 MB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Jul 4, 2025
- Updated
- Jul 10, 2025
- SHA
- 67d49cde
Try a prompt
Datasets
Model card
From the Hugging Face model README
XLM-RoBERTa Khmer Custom Tokenizer
This is a custom tokenizer trained for Khmer language based on XLM-RoBERTa architecture.
Model Details
- Model Type: XLM-RoBERTa Tokenizer
- Language: Khmer (km)
- Training Data: Custom Khmer dataset with ~84M examples
- Tokenizer Type: SentencePiece-based
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("metythorn/xlm-roberta-tokenizer-50k")
# Example usage
text = "ខ្ញុំចង់រៀនភាសាខ្មែរ"
tokens = tokenizer.tokenize(text)
print(tokens)
Training Details
- Base model: XLM-RoBERTa
- Training examples: 84Million
- Epochs: 3
- Batch size: 8
- Parameters: 278,295,186
Intended Use
This tokenizer is designed for:
- Khmer text processing
- Masked language modeling
- Fine-tuning on Khmer NLP tasks
- Research in Khmer language understanding
Limitations
- Primarily trained on Khmer text
- May not handle code-switching effectively
- Performance on formal vs informal Khmer may vary
- License
- apache-2.0
- Languages
- km
- Created
- Jul 4, 2025
- Updated
- Jul 10, 2025