EuroEval

gemma-3-tokenizer

EuroEval/gemma-3-tokenizer

gemma-3-tokenizer is a machine learning model from EuroEval. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.

This is the tokenizer used by the Gemma-3 models, and is used in the EuroEval testing suite. We're using this repo rather than the original ones since the original ones are gated, causing CI problems.