Solar-Open2-250B-W4A8

Solar-Open2-250B-W4A8

Weight-INT4 / activation-FP8 (W4A8) quantization of upstage/Solar-Open2-250B, self-baked and served in production on H200 GPUs. Only the routed MoE expert weights are quantized to INT4 (group size 128); the shared expert, dense MLP, attention projections, and embeddings stay bf16. KV cache is served in FP8.

• Recipe: OMSE (optimal mean-squared-error) calibration for the INT4 routed-expert weights — dominated a plain RTN (round-to-nearest) bake on the same accuracy gate, so OMSE is the shipped recipe.
• Footprint: 132 GiB total (vs. ~467 GiB bf16) — fits within TP4 on H200×4 minimum, TP8 recommended for headroom.
• Serving stack: vLLM fork (v0.26.0-solar-open2) with 3 upstream 0.22→0.26 porting-regression fixes applied to the hybrid-attention (KDA linear-attention) decode path — see Serving below.

Accuracy

Self-measured, same-question paired A/B against the bf16 original (frozen question sets, identical harness, temperature=0.6, max_tokens=40960):

Larger-sample backstop (same harness, un-paired):

Additional quality probes, both PASS: end-of-sequence behavior on 3 long-form generation prompts (finish_reason=stop 3/3, no runaway repetition), tool-calling correctness on 2 single/parallel-call probes (2/2 valid parsed calls). "raw" scores are pulled down mainly by max_tokens truncation on the hardest reasoning items (finish_reason=length) — a harness property shared with the bf16 anchor, not a quantization-specific effect; the conclusion-only column removes those unanswered items.

⚠️ These benchmarks run at a few-thousand-token input context. Long-input-context accuracy (production p50 context is far longer) has not been separately measured — quantization error can be context-length dependent, so treat this as a partial accuracy cover, not a full one.

Performance

In-house serving measurements on a fixed TP4×2 (8×H200) topology with an fp8-KV, prefix-caching engine, at an offered load near the goodput knee for this checkpoint (open-loop replay of production-shaped traffic, with a DSpark speculative-decoding draft attached — see Solar-Open2-250B-DSpark):

Absolute throughput (TPM) and node/GPU counts are withheld per internal reporting policy; the latency figures above are the release gate.

Serving

vLLM ≥ 0.26.0 with the Solar-Open2 fork. The hybrid-attention (12 softmax-attention layers interleaved with 36 KDA linear-attention layers) decode path needs three fixes ported from the 0.22.0 lineage — without them the model either corrupts state, mis-tracks speculative-decoding accept counts, or crashes on CUDA-graph capture. All three are baked into the served image; see the repo's serving notes if building your own image.

vllm serve vessl/Solar-Open2-250B-W4A8 \
  --served-model-name solar-open2-250b \
  --tensor-parallel-size 4 --enable-expert-parallel \
  --max-num-batched-tokens 65536 \
  --moe-backend triton \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --stream-interval 18 \
  --max-num-seqs 64 \
  --attention-config '{"flash_attn_max_num_splits_for_cuda_graph": 8}' \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --default-chat-template-kwargs '{"think_render_option":"preserved"}' \
  --reasoning-parser solar_open2 --tool-call-parser solar_open2 \
  --enable-auto-tool-choice \
  --port 8000

Required environment on multi-GPU-per-node hosts: VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm. Some hosts auto-select the mnnvl all-reduce backend, which initializes without error but produces far worse latency — check the boot log line Auto-selected flashinfer allreduce backend: ... and force trtllm if it isn't already selected.

--enable-expert-parallel is required regardless of tensor-parallel degree: the INT4 grouped-GEMM MoE kernel needs intermediate_size divisible into 256-wide partitions, which plain tensor parallelism can violate at some TP degrees; expert parallelism sidesteps the slicing entirely.

To attach the DSpark speculative-decoding draft:

--speculative-config '{"method":"dspark","model":"vessl/Solar-Open2-250B-DSpark","num_speculative_tokens":4}'

CUTLASS W4A8 MoE requires intermediate_size to be divisible by 256 per TP partition — this checkpoint is validated at TP4/EP4; a TP8 variant would need re-baking (or re-verification of the same divisibility constraint) before use.

License

This is a derivative of Solar Open 2 and is distributed under the same Upstage Solar License as the base model. Per the license's requirements for derivative AI models: this model's name is prefixed with "Solar," and this card carries a "Built with Solar" attribution. See the base model's license for full terms.

---

Original model card: upstage/Solar-Open2-250B

Solar Open 2

Solar Open 2 is Upstage's 250B-A15B open-weight large language model, built for agentic use cases such as office productivity, document-intensive work, and coding. Its Hybrid-Attention Mixture-of-Experts (MoE) architecture with linear attention delivers highly efficient inference even in long-context settings.

Technical Report | Blog | Upstage Website

Highlights

• Agentic Specialist: Purpose-built for agentic workflows — tool calling, multi-step reasoning, and end-to-end task execution. Competitive with the strongest open-weight models on agent benchmarks.

• Minimal Inference Cost: A 250B-parameter MoE that activates only 15B per token, built on a hybrid attention stack that interleaves three linear-attention layers with one softmax-attention layer.

• 1M-Token Context: The linear-attention layers encode token order intrinsically in their recurrent state, so positional encoding is removed entirely (NoPE), lifting the RoPE extrapolation limit.

• Efficiently Trained at Low Cost: Initialized by selective weight transfer from Solar Open 1 (102B) — only 2.3% of weights survive the architectural change.

• Multilingual: English, Korean, and Japanese.

---

Model Overview

---

Performance

English Benchmarks

Korean Benchmarks

---

Quickstart

Transformers

Install dependencies:

python -m pip install -U \
  "git+https://github.com/upstageAI/[email protected]" \
  "fla-core[cuda]>=0.5.1" \
  accelerate einops

Run the model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "upstage/Solar-Open2-250B"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=False,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    dtype=torch.bfloat16,
    trust_remote_code=False,
)
model.eval()

messages = [
    {"role": "user", "content": "What is Upstage?"},
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    reasoning_effort="high",
    think_render_option="preserved",
)
input_device = model.get_input_embeddings().weight.device
model_inputs = tokenizer(prompt, return_tensors="pt").to(input_device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=32768,
    do_sample=True,
    temperature=1.0,
    top_p=1.0,
)

new_token_ids = generated_ids[0, model_inputs.input_ids.shape[-1] :].tolist()
think_end_id = tokenizer.convert_tokens_to_ids("<|think:end|>")

if think_end_id in new_token_ids:
    answer_start = len(new_token_ids) - new_token_ids[::-1].index(think_end_id)
else:
    answer_start = len(new_token_ids)

reasoning = tokenizer.decode(
    new_token_ids[:answer_start],
    skip_special_tokens=True,
).strip()
answer = tokenizer.decode(
    new_token_ids[answer_start:],
    skip_special_tokens=True,
).strip()

print("[reasoning]", reasoning)
print("[answer]", answer)

Serving with vLLM (Recommended)

Option 1: Docker

docker run --rm --gpus all --ipc=host \
  -p 8000:8000 \
  -v "${HF_HOME:-$HOME/.cache/huggingface}:/root/.cache/huggingface" \
  upstage/vllm-solar-open2 \
  upstage/Solar-Open2-250B \
  --served-model-name solar-open2-250b \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend triton \
  --default-chat-template-kwargs '{"think_render_option":"preserved"}' \
  --reasoning-parser solar_open2 \
  --tool-call-parser solar_open2 \
  --enable-auto-tool-choice \
  --logits-processors vllm.v1.sample.logits_processor.solar_open2:SolarOpen2TemplateLogitsProcessor

Option 2: Install from source

pip install -U uv

VLLM_PRECOMPILED_WHEEL_LOCATION="https://github.com/vllm-project/vllm/releases/download/v0.22.0/vllm-0.22.0%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl" \
VLLM_USE_PRECOMPILED=1 \
uv pip install --reinstall-package vllm --torch-backend=cu129 \
  "git+https://github.com/UpstageAI/[email protected]"

Start the server:

vllm serve upstage/Solar-Open2-250B \
  --served-model-name solar-open2-250b \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend triton \
  --default-chat-template-kwargs '{"think_render_option":"preserved"}' \
  --reasoning-parser solar_open2 \
  --tool-call-parser solar_open2 \
  --enable-auto-tool-choice \
  --logits-processors vllm.v1.sample.logits_processor.solar_open2:SolarOpen2TemplateLogitsProcessor

Send a chat completion request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "solar-open2-250b",
    "messages": [
      {"role": "user", "content": "What is Upstage?"}
    ],
    "max_tokens": 131584,
    "temperature": 1.0,
    "top_p": 1.0,
    "reasoning_effort": "high"
  }'

Quantized Versions

Official quantized models by NotaAI are available:

• Solar-Open2-250B-Nota-INT4-GlobalPruned
• Solar-Open2-250B-Nota-NVFP4
• Solar-Open2-250B-Nota-INT4

---

Capabilities

Reasoning

Use reasoning_effort="high" for reasoning and reasoning_effort="none" for a direct response. The recommended vLLM configuration limits a reasoning block to 131,072 tokens.

Example:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="solar-open2-250b",
    messages=[
        {
            "role": "user",
            "content": "Prove that the square root of 2 is irrational.",
        },
    ],
    reasoning_effort="high",
    temperature=1.0,
    top_p=1.0,
    max_tokens=131584,
)

print(response.choices[0].message.reasoning)
print(response.choices[0].message.content)

Tool Calling

Tool calls follow the standard OpenAI function-calling interface. Start the server with --tool-call-parser solar_open2 and --enable-auto-tool-choice.

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                },
                "required": ["location"],
            },
        },
    },
]

response = client.chat.completions.create(
    model="solar-open2-250b",
    messages=[
        {
            "role": "user",
            "content": "What's the weather in Seoul?",
        },
    ],
    tools=tools,
)

print(response.choices[0].message.tool_calls)

---

Agentic Use

Both Anthropic's Claude Code and Nous Research's Hermes Agent can run on Solar Open 2 served locally with vLLM. A single vLLM server exposes both interfaces.

Claude Code

vLLM exposes an Anthropic-compatible /v1/messages endpoint:

export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_AUTH_TOKEN=dummy
export ANTHROPIC_MODEL=solar-open2-250b
export ANTHROPIC_SMALL_FAST_MODEL=solar-open2-250b
claude

Hermes Agent

Register the local vLLM server in ~/.hermes/config.yaml:

model:
  provider: custom
  default: solar-open2-250b
  base_url: http://localhost:8000/v1
  api_key: dummy

---

Best Practices

Recommended client-side generation settings

Solar Open 2 is a reasoning-capable model. Use reasoning_effort="high" for complex or agentic tasks.

Recommended settings by reasoning mode

Key recommendations:

• Set max_tokens high enough (up to 256K) — reasoning traces can be long and may otherwise truncate the answer.
• The reasoning trace is preserved by default (think_render_option=preserved).
• Multi-turn: Keep prior reasoning traces in the conversation history. The default think_render_option=preserved handles this automatically.
• Parsing: the OpenAI-compatible server returns reasoning in a separate message.reasoning field; with local transformers, split the raw output on the reasoning markers.

---

License

Solar Open 2 is distributed under the Upstage Solar License.

Key requirements for Derivative AI Models:

• Naming: prefix your model name with "Solar" (e.g., Solar-MyModel-v1).
• Attribution: prominently display "Built with Solar" in related public-facing materials.
• Notice: include a copy of the Upstage Solar License with your derivative model.

---

Citation

@misc{solaropen2-2026,
    title={Solar Open 2 Technical Report},
    author={Upstage AI},
    year={2026},
    eprint={2607.20062},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2607.20062}
}