byt5-small-multilingual-4g

hmbyt5-preliminary/byt5-small-multilingual-4g

byt5-small-multilingual-4g is a machine learning model from hmbyt5-preliminary. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.

Preliminary Historic Multilingual and Monolingual ByT5 Models. Following languages are currently covered:

Downloads · 30 days

2% of all-time downloads

All-time downloads

719

Public

Parameters

300M

21.8 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.bin1.2 GB · 33%

At a glance

Library: transformers
License: mit
Model type: t5
Access: Public
Created: Apr 10, 2023
Updated: Nov 19, 2023
SHA: 40d7a9a4

Model card

From the Hugging Face model README

hmByT5 - Preliminary Language Models

Preliminary Historic Multilingual and Monolingual ByT5 Models. Following languages are currently covered:

English (British Library Corpus - Books)
German (Europeana Newspaper)
French (Europeana Newspaper)
Finnish (Europeana Newspaper)
Swedish (Europeana Newspaper)
Dutch (Delpher Corpus)

More details can be found in our GitHub repository.

In this experiment we sample 4B bytes (~4GB of text) from each corpora (and upsample Swedish and Finnish).

Pretraining

We use the official JAX/FLAX example in Hugging Face Transformers to pretrain a ByT5 model on a single v3-8 TPU. Details about the training can be found here.

Evaluation on Downstream Tasks (NER)

We evaluated the hmByT5 model on downstream tasks:

Model	English AjMC	German AjMC	French AjMC	Finnish NewsEye	Swedish NewsEye	Dutch ICDAR	French ICDAR	Avg.
`hmbyt5-preliminary/byt5-small-multilingual-4g`	83.49 ± 0.96	87.65 ± 0.63	84.16 ± 0.90

Acknowledgements

Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Many Thanks for providing access to the TPUs ❤️

Library: transformers
Type: t5
License: mit
Languages: en, de, fr, fi, sv, nl
Created: Apr 10, 2023
Updated: Nov 19, 2023