functionalizer-100M-fineweb-edu-seed3

Functionalizer 100M model

This is a 100M example model. Each model is trained on the dataset specified in the name for 50000 epochs with the given seed.

This model is presented in the paper The Functionalizer: Lossless Functional Decomposition for Subword Tokenization.

The model was trained using the Functionalizer framework. Training code and detailed performance analysis are available: https://github.com/connor-makowski/functionalizer

Running the model

To run the model, use the custom tokenizer fork available here: https://github.com/connor-makowski/tokenizers/tree/functionalizer

Citation

If you find this model or the Functionalizer framework useful, please cite:

@misc{makowski2026functionalizerlosslessfunctionaldecomposition,
      title={The Functionalizer: Lossless Functional Decomposition for Subword Tokenization}, 
      author={Connor Makowski and Willem Guter},
      year={2026},
      eprint={2609.15991},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.15991}, 
}