Downloads · 30 days
0
icebreaker-tokenicer
Sigurdur/icebreaker-tokenicer
icebreaker-tokenicer is a machine learning model from Sigurdur. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
This is a BPE tokenizer trained on the Iceladic Gigaword Corpus, News 1. The tokenizer can be used for training Icelandic language models.
Access
Public
Updated Mar 8, 2024
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json2.1 MB · 85%
At a glance
- Library
- transformers
- Access
- Public
- Created
- Mar 5, 2024
- Updated
- Mar 8, 2024
- SHA
- 4e981b10
Model card
From the Hugging Face model README
Icebreaker tokenizer
This is a BPE tokenizer trained on the Iceladic Gigaword Corpus, News 1. The tokenizer can be used for training Icelandic language models.
Model Details
BPE tokenizer, trained on the first 242553 files in the News 1 IGC 2022, unnanotated dataset by Arnastofnun.
Model Description
It has a vocab size of 3200.
- Developed by: Sigurdur Haukur Birgisson
- Model type: GPT2Tokenizer
- Language(s) (NLP): Icelandic
Model Sources
- Repository: https://github.com/sigurdurhaukur/tokenicer
How to Get Started with the Model
Use the code below to get started with the model.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Sigurdur/icebreaker")
tokens = tokenizer("Halló heimur!")
Model Card Contact
Sigurdur Haukur Birgissson: [email protected]
- Library
- transformers
- Languages
- is
- Created
- Mar 5, 2024
- Updated
- Mar 8, 2024