Downloads · 30 days
43
3% of all-time downloads
byt5-small-historic-multilingual-span20-flax
hmbyt5-preliminary/byt5-small-historic-multilingual-span20-flax
byt5-small-historic-multilingual-span20-flax is a machine learning model from hmbyt5-preliminary. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
Preliminary Historic Multilingual and Monolingual ByT5 Models. Following languages are currently covered:
All-time downloads
1.5K
Public
Repo size
109 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin1.2 GB · 47%
At a glance
- Library
- transformers
- License
- mit
- Model type
- t5
- Access
- Public
- Created
- Apr 29, 2023
- Updated
- Oct 27, 2023
- SHA
- aa3c4083
Model card
From the Hugging Face model README
hmByT5 - Preliminary Language Models
Preliminary Historic Multilingual and Monolingual ByT5 Models. Following languages are currently covered:
- English (British Library Corpus - Books)
- German (Europeana Newspaper)
- French (Europeana Newspaper)
- Finnish (Europeana Newspaper)
- Swedish (Europeana Newspaper)
- Dutch (Delpher Corpus)
More details can be found in our GitHub repository.
Pretraining
We use the official JAX/FLAX example in Hugging Face Transformers to pretrain a ByT5 model on a single v3-8 TPU. Details about the training can be found here.
This model was trained with mean_noise_span_length=20 for one epoch.
Evaluation on Downstream Tasks (NER)
See detailed results at hmLeaderboard.
Acknowledgements
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs ❤️
- Library
- transformers
- Type
- t5
- License
- mit
- Languages
- en, de, fr, fi, sv, nl
- Created
- Apr 29, 2023
- Updated
- Oct 27, 2023