SIRL-Qwen3-8B

SIRL-Qwen3-8B

SIRL (Structural-Informed Reinforcement Learning) fine-tunes Qwen3-8B to translate natural language descriptions into mathematical optimization models (Gurobi Python API).

The model is trained with a structure-aware reward that evaluates not only the final objective value, but also the structural fidelity of the generated model — variable counts, constraint cardinalities, binary/integer variable ratios, and quadratic term matching.

Key Results (shot prompt, pass@1)

The largest gain is on MAMO Complex (challenging mixed-integer programs), where structural reward signals provide a +8.4% improvement over the base model.

How It Works

Structural Reward

Instead of only checking whether the final objective value matches, SIRL uses a multi-component reward:

score = ans_ok × 1.0 + code_ok × 1.0 + format × 0.5 + lp_score × 0.75
max   = 3.25

• ans_ok: does the generated model produce the correct objective value?
• code_ok: is the generated code executable (no syntax/API errors)?
• format: does the output follow the required structure?
• lp_score: how closely do the variable/constraint counts match the ground-truth model?

The lp_score component measures structural alignment across:

• Objective type (min/max)
• Number of variables and constraints
• Number of binary and integer variables
• Presence of quadratic terms

Training

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "theGuo/SIRL-Qwen3-8B",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("theGuo/SIRL-Qwen3-8B")

messages = [
    {"role": "system", "content": "You are an expert in mathematical optimization..."},
    {"role": "user", "content": "A company produces two products..."},
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=3072, temperature=0.0)
print(tokenizer.decode(outputs[0]))

The model is trained to output Gurobi Python code in <python>...</python> tags.

Prompt Format

The model was trained with a few-shot prompt format (no <think> tags). For best results, use the shot-style system prompt that matches the training distribution.

Citation

@misc{sirl2026,
  title={SIRL: Structural-Informed Reinforcement Learning for Optimization Modeling},
  author={Guo, ...},
  year={2026},
}

License

Apache 2.0 (same as Qwen3-8B).