qwen3_80M_token_2B_lr_5e-3_bsz_16_seed_0

Model Details

The model is trained on optimization trajectories of blackbox optimization methods evaluated on different optimization tasks. It's based on a Qwen3-type architecture. More details are described in the paper:

An Open-Source Training Dataset for Foundation Models for Black-box Optimization Aaron Klein, Herilalaina Rakotoarison, Luca Thale-Bombien, David Salinas arXiv:2605.23417 [cs.LG]

Downstream Use

For an example how to use this model for optimization see the Syne-Tune library.

Training Details

Training Data

The model is trained on the BBO-Pile dataset.

Training Procedure

We used AdamW with β1 = 0.9, β2 = 0.95, weight_decay = 0.1, gradient clipping with max_norm = 1.0, and a cosine learning rate schedule with 10% linear warm-up. All models are trained with bf16-mixed, without gradient accumulation, and a context length of 4096 tokens

Citation

If you want to use this model please cite the following paper:

BibTeX:

@article{bbo-pile2026, title={An Open-Source Training Dataset for Foundation Models for Black-box Optimization}, author={Aaron Klein and Herilalaina Rakotoarison and Luca Thale-Bombien and David Salinas}, year={2026}, journal={arXiv:2605.23417 [cs.LG]} }