Solar-Open-100B-AWQ-8bit

<div align="left"> <a href="https://cyan.kiwi" target="_blank"><img src="https://storage.googleapis.com/cyankiwi-assets/model-card/model-card-banners-cyankiwi.png" width="800" alt="cyankiwi, the LLM inference agent for open models"></a> </div> <div align="left"> <table style="border-collapse:collapse; border:none;"> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Website</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://cyan.kiwi/" target="_blank">cyan.kiwi</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Models</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://huggingface.co/cyankiwi" target="_blank">Hugging Face</a> · <a href="https://www.modelscope.ai/organization/cyankiwi" target="_blank">ModelScope</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Connect</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://x.com/cyankiwi_ai" target="_blank">X</a> · <a href="https://www.reddit.com/r/cyankiwi/" target="_blank">Reddit</a> · <a href="mailto:[email protected]">Email</a></td> </tr> </table> </div> <br> <div align="left"> <img src="https://storage.googleapis.com/cyankiwi-assets/model-card/model-card-banners-awq.png" width="800" alt="cyankiwi Activation-aware Weight Quantization"> </div> <div align="left"> <table style="border-collapse:collapse; border:none;"> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Version</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://github.com/vllm-project/llm-compressor" target="_blank">llm-compressor</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Calibration</b></td> <td align="left" style="border:none; padding:4px 0;"><a href="https://huggingface.co/datasets/nvidia/Llama-Nemotron-Post-Training-Dataset" target="_blank">nvidia/Llama-Nemotron-Post-Training-Dataset</a></td> </tr> <tr style="border:none;"> <td align="left" style="border:none; padding:4px 24px 4px 0;"><b>Model Size</b></td> <td align="left" style="border:none; padding:4px 0;">111.32 GB</td> </tr> </table> </div> <br> <p align="left"> <a href="https://cyan.kiwi/workspace?prompt=Deploy%20cyankiwi/Solar-Open-100B-AWQ-8bit" target="_blank"><img src="https://storage.googleapis.com/cyankiwi-assets/model-card/model-card-button-deploy.png" width="333" alt="Deploy on cyan.kiwi private endpoints"></a> </p>

---

<p align="center"> <img src="./Solar-Open-100B.png" alt="Solar Open Model" width="100%"> </p>

Solar Open

Solar Open is Upstage's flagship 102B-parameter large language model, trained entirely from scratch and released under the Upstage Solar License (see LICENSE for details). As a Mixture-of-Experts (MoE) architecture, it delivers enterprise-grade performance in reasoning, instruction-following, and agentic capabilities—all while prioritizing transparency and customization for the open-source community.

Technical Report | Project Page

Highlights

• MoE Architecture (102B / 12B): Built on a Mixture-of-Experts architecture with 102B total / 12B active parameters. This design delivers the knowledge depth of a massive model with the inference speed and cost-efficiency of a much smaller model.
• Massive Training Scale: Pre-trained on 19.7 trillion tokens, ensuring broad knowledge coverage and robust reasoning capabilities across various domains.
• Quantized Version Available: An official INT4 quantized model is provided by NotaAI and available at nota-ai/Solar-Open-100B-NotaMoEQuant-Int4.

Model Overview

• Model Name: Solar Open 100B
• Hugging Face ID: Upstage/Solar-Open-100B
• Architecture: Mixture-of-Experts (MoE)
  • Total Parameters: 102.6B
  • Active Parameters: 12B (per token)
  • Experts: 129 Experts (top 8 among 128 Routed + 1 Shared)
• Pre-training Tokens: 19.7 Trillion
• Context Length: 128k
• Training Hardware: NVIDIA B200 GPUs
• License: Upstage Solar License (See LICENSE)
• Hardware Requirements:
  • Minimum: 4x NVIDIA A100 (80GB)

For more details, please refer to the Solar Open Technical Report.

License

This repository contains both model weights and code, which are licensed under different terms:

1. MODEL WEIGHTS (*.safetensors) Licensed under Upstage Solar License See: https://huggingface.co/upstage/Solar-Open-100B/blob/main/LICENSE

2. CODE (*.py, *.json, *.jinja files) Licensed under Apache License 2.0 See: https://www.apache.org/licenses/LICENSE-2.0

Performance

Korean Benchmarks

English Benchmarks

Inference Quickstart

We recommend using the following generation parameters:

temperature=0.8
top_p=0.95
top_k=50

Transformers

Install the required dependencies:

pip install -U "transformers>=5.0" kernels torch accelerate

Run inference with the following code:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "upstage/Solar-Open-100B"

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

model = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path=MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Prepare input
messages = [{"role": "user", "content": "who are you?"}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
)
inputs = inputs.to(model.device)

# Generate response
generated_ids = model.generate(
    **inputs,
    max_new_tokens=4096,
    temperature=0.8,
    top_p=0.95,
    top_k=50,
    do_sample=True,
)
generated_text = tokenizer.decode(generated_ids[0][inputs.input_ids.shape[1] :])
print(generated_text)

vLLM

Option 1: Using Docker (Highly Recommended)

Docker is the recommended deployment method for running Solar-Open-100B.

# For 8 GPUs
docker run --gpus all \
    --ipc=host \
    -p 8000:8000 \
    upstage/vllm-solar-open:latest \
    upstage/Solar-Open-100B \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser solar_open \
    --reasoning-parser solar_open \
    --logits-processors vllm.model_executor.models.parallel_tool_call_logits_processor:ParallelToolCallLogitsProcessor \
    --logits-processors vllm.model_executor.models.solar_open_logits_processor:SolarOpenTemplateLogitsProcessor \
    --tensor-parallel-size 8

Option 2: Installing from Source

For development, debugging, custom modifications or offline inference, Solar Open can also be run using a source installation of vLLM. We recommend using uv for environment management and dependency resolution.

Create and activate a Python virtual environment

uv venv --python 3.12 --seed
source .venv/bin/activate

Install Solar Open's optimized vLLM

VLLM_PRECOMPILED_WHEEL_LOCATION="https://github.com/vllm-project/vllm/releases/download/v0.12.0/vllm-0.12.0-cp38-abi3-manylinux_2_31_x86_64.whl" \
VLLM_USE_PRECOMPILED=1 \
uv pip install git+https://github.com/UpstageAI/[email protected]

Start the vLLM server (For 8 GPUs)

vllm serve upstage/Solar-Open-100B \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser solar_open \
    --reasoning-parser solar_open \
    --logits-processors vllm.model_executor.models.parallel_tool_call_logits_processor:ParallelToolCallLogitsProcessor \
    --logits-processors vllm.model_executor.models.solar_open_logits_processor:SolarOpenTemplateLogitsProcessor \
    --tensor-parallel-size 8

Citation

If you use Solar Open in your research, please cite:

@article{park2025solar,
  title={Solar Open Technical Report},
  author={Sungrae Park and Sanghoon Kim and Jungho Cho and Gyoungjin Gim and Dawoon Jung and Mikyoung Cha and Eunhae Choo and Taekgyu Hong and Minbyul Jeong and SeHwan Joo and Minsoo Khang and Eunwon Kim and Minjeong Kim and Sujeong Kim and Yunsu Kim and Hyeonju Lee and Seunghyun Lee and Sukyung Lee and Siyoung Park and Gyungin Shin and Inseo Song and Wonho Song and Seonghoon Yang and Seungyoun Yi and Sanghoon Yoon and Jeonghyun Ko and Seyoung Song and Keunwoo Choi and Hwalsuk Lee and Sunghun Kim and Du-Seong Chang and Kyunghyun Cho and Junsuk Choe and Hwaran Lee and Jae-Gil Lee and KyungTae Lim and Alice Oh},
  journal={arXiv preprint arXiv:2601.07022},
  year={2025},
  url={https://huggingface.co/papers/2601.07022}
}