Downloads · 30 days
31
2% of all-time downloads
byt5-small-historic-multilingual
hmbyt5-preliminary/byt5-small-historic-multilingual
byt5-small-historic-multilingual is a machine learning model from hmbyt5-preliminary. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
Preliminary Historic Multilingual and Monolingual ByT5 Models. Following languages are currently covered:
All-time downloads
1.4K
Public
Parameters
300M
9.9 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.data-00001-of-000041.6 GB · 21%
At a glance
- Library
- transformers
- License
- mit
- Model type
- t5
- Access
- Public
- Created
- Apr 4, 2023
- Updated
- Jun 17, 2025
- SHA
- 1596f31d
Model card
From the Hugging Face model README
hmByT5 - Preliminary Language Models
Preliminary Historic Multilingual and Monolingual ByT5 Models. Following languages are currently covered:
- English (British Library Corpus - Books)
- German (Europeana Newspaper)
- French (Europeana Newspaper)
- Finnish (Europeana Newspaper)
- Swedish (Europeana Newspaper)
- Dutch (Delpher Corpus)
More details can be found in our GitHub repository.
Pretraining
We pretrain hmByT5 on a v3-32 TPU Pod. Details about the training can be found here.
Evaluation on Downstream Tasks (NER)
We evaluated the hmByT5 model that was pretrained on English AjMC corpus for 200k steps:
| Hyper-param Configuration | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Avg. |
|---|---|---|---|---|---|---|
wsFalse-bs4-e10-lr0.00016-poolingfirst | 83.80 | 84.78 | 83.74 | 83.35 | 84.37 | 84.01 ± 0.50 |
wsFalse-bs4-e10-lr0.00015-poolingfirst | 84.67 | 82.69 | 83.92 | 84.53 | 82.90 | 83.74 ± 0.82 |
wsFalse-bs8-e10-lr0.00016-poolingfirst | 82.12 | 83.82 | 83.37 | 83.00 | 83.70 | 83.20 ± 0.61 |
wsFalse-bs8-e10-lr0.00015-poolingfirst | 83.45 | 82.83 | 84.15 | 81.76 | 83.78 | 83.19 ± 0.84 |
It turns out, that the results are not on-par with current SOTA on the English AjMC corpus, see a comparison here. Thus, we continue experiments with the Hugging Face Transformers JAX/FLAX implementation to pretrain ByT5 models on TPU.
Acknowledgements
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs ❤️
- Library
- transformers
- Type
- t5
- License
- mit
- Languages
- en, de, fr, fi, sv, nl
- Created
- Apr 4, 2023
- Updated
- Jun 17, 2025