Qwen3-SIRL-4B

<h2 align="center"> OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling</h2> <p align="center"> <a href="https://icml.cc/virtual/2026/poster/61027"><img src='https://img.shields.io/badge/ICML-2026-007ACC?logo=icloud&logoColor=white'></a> <a href="https://arxiv.org/abs/2601.19924"><img src='https://img.shields.io/badge/arXiv-2601.19924-b31b1b?logo=arxiv&logoColor=white'></a> <a href="https://huggingface.co/chenyitian-shanshu/Qwen3-SIRL-4B"><img src='https://img.shields.io/badge/🤗%20HuggingFace-Model-yellow'></a> </p>

Overview

We introduce OPT-Engine, an extensible benchmark framework for optimization problems with controllable complexity and configurable templates. OPT-Engine spans ten canonical operations research problem classes,systematically scaling in complexity, thus provides a structured testbed for automated problem formulation and solving under different OR complexity. OPT-Engine facilitates rigorous, reproducible studies on how problem complexity impacts model performance, offering a more granular look at LLM formulation and solving capabilities.

1. First, we examine whether Pure-Text Reasoning (PTR) via classical Chain-of-Thought can efficiently tackle optimization tasks, finding that PTR suffers from a critical robustness gap as task complexity increases.
2. Second, we examine whether integrating external computational tools can mitigate PTR's arithmetic weaknesses and improve performance. Our results indicate that while such tools help with local calculations, they still fail to adhere to global optimization constraints.
3. Finally, we pinpoint that for the current SOTA paradigm, Solver-integrated Reasoning (SIR), the automated formulation of constraints represents the primary bottleneck.

Updates

• 2026.05.01 - Our paper has been accepted for a poster presentation at ICML 2026! 🔥
• 2026.01.29 - OPT-Engine paper published on arXiv: OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling.

The OPT-Engine Pipeline

For a given problem class, the OPT-Engine pipeline operates as follows:

1. Numeric Instance Generation & Validation: It first samples a random numerical instance within a specified target complexity range, defined by parameters such as variable and constraint counts. A Gurobi-based exact solver then verifies the instance's feasibility and computes the optimal solution, which is stored as the ground truth.
2. Canonical Problem Creation: Once validated, the framework maps the instance's numeric parameters into a structured, editable template to produce a canonical natural-language problem statement.
3. Problem augumentation: To decouple linguistic variation from mathematical structure, the pipeline employs an LLM-based rephrasing step. This step alters the textual scenario and surface wording while strictly preserving the objective function, constraints, and all numerical values.
4. Integrity Verification: A subsequent LLM-as-a-judge and rule-based validation check uses regex extraction to confirm that the rephrased text maintains the original numeric parameters and their logical relationships. If validation fails, the rephrasing step is retried until a valid output is produced.

C^3-Bench Dataset

We also provide C^3-Bench (Controllable & Configurable Complexity Benchmark), a dataset generated by our framework to facilitate reproducible research and enable further studies. The dataset is hosted alongside the complete OPT-Engine benchmark suite:

1. C³-Bench Dataset (source data): Available on GitHub: C^3-Bench Dataset.
2. OPT-Engine Benchmarks (ready-to-use): Hosted on Hugging Face Datasets for easy loading via the datasets library: huggingface Dataset.

C³-Bench is structured into two complementary subsets for evaluating model performance on classical Operations Research (OR) problems under controlled complexity scaling:

1. canonical: Contains standard instances of 10 canonical OR problem types, with complexity systematically increased across parameters like variable and constraint counts.
2. perturbation: Contains derived instances where controlled linguistic and parametric perturbations are applied to a subset of the canonical benchmark (covering the Inventory, TSP, and Knapsack problems) to specifically test model robustness and generalization.

Within the perturbation set, we introduce controlled variations across three dimensions that are known performance bottlenecks:

• Linguistic Complexity: The underlying mathematical model is held constant, while the natural language description is rephrased into templates with systematically higher syntactic and lexical complexity, creating harder-to-parse problem statements.

• Objective Perturbation: A constant term or simple coefficient change is introduced to the objective function, testing the model's ability to adapt to modified goals while the constraints remain unchanged.

• Constraint Augmentation: One additional simple linear constraint is introduced to the original formulation, testing how models handle incremental increases in problem structure without altering the core variables or objective.

Inference

We recommend using the following S.I.R prompt template which can be found in benchmark_gurobi_prompts.py. Please replace the {question} with any natural language OR question.

Quick start

Below is a simple example for model inference:

from transformers import AutoTokenizer
from  benchmark_prompt_utils import benchmark_gurobi_prompts
from utils import extract_code_block, extract_obj
from vllm import SamplingParams, LLM
from langchain_core.prompts import PromptTemplate
import subprocess

# Load model and parameters
model = LLM("chenyitian-shanshu/Qwen3-SIRL-4B",            
            tensor_parallel_size=1,
            trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("chenyitian-shanshu/Qwen3-SIRL-4B")
sampling_params = SamplingParams(
            n=1,
            temperature=0.5,
            top_p=0.95,
            max_tokens=8192,
            repetition_penalty=1.02
        )

# Load question. Here is just an example. Users can replace this with datasets they want to test
question = "An industrial tire company delivers large tires for equipment to remote engineering sites either by cargo planes or ultrawide trucks. Each cargo plane can transport 10 tires per trip and costs $1000. Each ultrawide truck can transport 6 tires per trip and costs $700. The company needs to transport at least 200 tires and has available $22000. Because most remote sites don't have proper airports, the number of plane trips cannot exceed the number of ultrawide truck trips. How many trips of each should be done to minimize the total number of trips?"

# Load prompt templete
TIR_prompt_user = PromptTemplate.from_template(benchmark_gurobi_prompts['zeroshot_q2mc_en'])
prompt =[ {"role": "user",
          "content": TIR_prompt_user.format(Question=question).strip() }]

# Generate Response
text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)
response = model.generate(text,sampling_params)
response_text = response[0].outputs[0].text
code_snippet = extract_code_block(response_text,'gurobi')
result = subprocess.run(['python3', '-c', code_snippet], capture_output=True, text=True, timeout=100)
obj = extract_obj(result.stdout,'gurobi')
print(response_text)
print('optimal value is', obj)

Test Dataset

We evaluate the performance of our trained model on multiple datasets which include MAMO, IndustryOR, OptMATH and the C^3-Bench Dataset(OPT-Engine datasets) from https://github.com/Cardinal-Operations/OPTEngine.

Citation

If you find OPT-Engine (https://github.com/Cardinal-Operations/OPTEngine) useful or relevant to your research, please consider citing our paper:

@article{chen2026opt,
  title={OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling},
  author={Chen, Yitian and Cheng, Cheng and Sun, Yinan and Ling, Zi and Ge, Dongdong},
  journal={arXiv preprint arXiv:2601.19924},
  year={2026}
}