NeoMME-260M-Pretrain-predecay-s450000

<p align="left"> <img src="https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/assets/neomme/neomme_logo.webp?raw=true" alt="NeoMME logo" style="max-height: 140px;"> </p>

NeoMME-260M predecay checkpoint

Model summary

This repository contains the NeoMME-260M checkpoint at step 450,000 of a 500,000-step pretraining run. As the model name suggests, this checkpoint was saved just before the learning rate decay phase. It is intended as a starting point for continued pretraining with NeoMMEForMaskedLM.

<table> <thead> <tr style="background-color: rgba(146, 81, 247, 0.20);"><th>Item</th><th>Value</th></tr> </thead> <tbody> <tr><td>Training step</td><td>450,000 of 500,000</td></tr> <tr><td>Intended use</td><td>Continued pretraining</td></tr> <tr><td>Model class</td><td><code>NeoMMEForMaskedLM</code></td></tr> <tr><td>Training objective</td><td>Predict masked text tokens from visible text and image patches</td></tr> </tbody> </table>

[!IMPORTANT] training_state.pt contains the original native NeoMME optimizer and training state. It is provided for research and custom continuation workflows, but it is not directly compatible with Hugging Face Trainer or standard AdamW. The file uses NeoMME's custom NorMuon and MasterAdamW state formats, native parameter grouping, and FP32 master weights.

Continue pretraining

The original run used the following optimizer settings:

<table> <thead> <tr style="background-color: rgba(146, 81, 247, 0.20);"><th>Parameter group</th><th>Optimizer</th><th>Peak learning rate</th><th>Weight decay</th></tr> </thead> <tbody> <tr><td>Two-dimensional weight matrices</td><td>NorMuon</td><td align="right">0.012</td><td align="right">0.01</td></tr> <tr><td>Embedding tables</td><td>AdamW</td><td align="right">0.0013</td><td align="right">0.01</td></tr> <tr><td>One-dimensional parameters</td><td>AdamW</td><td align="right">0.0013</td><td align="right">0</td></tr> </tbody> </table>

The original learning-rate schedule used:

• A 300-step linear warmup.
• A stable phase through step 450,000.
• A 50,000-step linear decay from the peak learning rate to 1% of the peak learning rate.

For a Transformers continuation with new optimizers, initialize them at the peak learning rates and apply the 50,000-step linear decay.

License

Model weights are released under the Apache 2.0 license.

Citation

@misc{lac2026neommesingletowermultimodalnativemultilingual,
      title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},
      author={Aurélien Lac and Tony Wu},
      year={2026},
      eprint={2609.01657},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2609.01657},
}