Downloads · 30 days
0
arabic_bpe_32k
tarekeldeeb/arabic_bpe_32k
arabic_bpe_32k is a machine learning model from tarekeldeeb. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
Arabic BPE Tokenization Using Google Sentance Piece.
Access
Public
Updated Mar 23, 2023
Repo size
918 KB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.model918 KB · 59%
At a glance
- License
- other
- Access
- Public
- Created
- Mar 23, 2023
- Updated
- Mar 23, 2023
- SHA
- f385cafa
Model card
From the Hugging Face model README
Arabic BPE Tokenization Using Google Sentance Piece.
Natural Language Processing is a branch of AI. One of the first steps in any NLP system is language model encoding. The challenge is how to present/encode the words efficiently. Sub-word encoding is very suitable to arabic. For example the word مدرساتهم will not be considered a single token/word, but split into three; مدرس, ات, and هم. This is the basic intuition. This process is done automatically without any rules or preprocessing.
Vocab size: 32000
Project: https://github.com/tarekeldeeb/arabic_byte_pair_encoding
License: Waqf v2
- License
- other
- Languages
- ar
- Created
- Mar 23, 2023
- Updated
- Mar 23, 2023