Downloads · 30 days
71
56% of all-time downloads
byt5-base-historic-dutch
hmbyt5/byt5-base-historic-dutch
byt5-base-historic-dutch is a machine learning model from hmbyt5. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
Preliminary Historic Multilingual and Monolingual ByT5 Models. Following languages are currently covered:
All-time downloads
126
Public
Repo size
38.1 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin2.3 GB · 49%
At a glance
- Library
- transformers
- License
- mit
- Model type
- t5
- Access
- Public
- Created
- May 15, 2023
- Updated
- May 24, 2023
- SHA
- 9d1e054f
Model card
From the Hugging Face model README
hmByT5 - Preliminary Language Models
Preliminary Historic Multilingual and Monolingual ByT5 Models. Following languages are currently covered:
- Dutch (Delpher Corpus)
More details can be found in our GitHub repository.
Pretraining
We use the official JAX/FLAX example in Hugging Face Transformers to pretrain a ByT5 model on a single v3-8 TPU. Details about the training can be found here.
Evaluation on Downstream Tasks (NER)
We evaluated the hmByT5 model on ICDAR Europeana dataset:
| Configuration | Run 1 | Avg. |
|---|---|---|
wsFalse-bs4-e10-lr0.00015-poolingfirst | 87.63 | 87.63 ± 0.0 |
wsFalse-bs8-e10-lr0.00016-poolingfirst | 87.35 | 87.35 ± 0.0 |
wsFalse-bs8-e10-lr0.00015-poolingfirst | 87.26 | 87.26 ± 0.0 |
wsFalse-bs4-e10-lr0.00016-poolingfirst | 86.31 | 86.31 ± 0.0 |
We only performed fine-tuning for one epoch. Unfortunately, this ByT5 Base model shows no improvement over ByT5 Small architecture.
Acknowledgements
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs ❤️
- Library
- transformers
- Type
- t5
- License
- mit
- Languages
- nl
- Created
- May 15, 2023
- Updated
- May 24, 2023