RakutenAI-3.0-W4A16

RakutenAI-3.0-W4A16

Weight-only quantization of Rakuten/RakutenAI-3.0, a 671B DeepSeek-V3 architecture MoE model.

• Routed experts: symmetric INT4 weights with group size 32.
• Attention, shared experts, the first three dense layers, and lm_head: BF16 weights.
• Activations: BF16.
• KV cache: not quantized by the checkpoint metadata.
• Checkpoint format: compressed-tensors 0.18.0, pack-quantized.
• INT4 selection method: round-to-nearest (RTN), with no calibration data.

Why this release uses RTN

W4A16 is the precision layout: 4-bit routed-expert weights and BF16 activations. It does not imply GPTQ. RTN, GPTQ, and AWQ-lite are methods that select the stored INT4 values. The three comparison checkpoints use the same source weights, quantized scope, group size, packing code, shard layout, prompts, targets, and B300 runtime settings. MMLU-Pro and BBH identities matched question by question.

The GPTQ control was lower than RTN on both large benchmarks. Its paired two-sided exact tests gave p=0.6131 for MMLU-Pro and p=0.02611 for BBH. The AWQ-lite control was lower on MMLU-Pro and higher on BBH. Its paired tests did not show an advantage over RTN (p=0.3462 and p=0.3647). AIME has only 30 questions and did not decide the method. This release uses RTN because neither calibrated method showed a stable accuracy advantage.

Validation

The checkpoint contains 63 weight shards, 134,483 indexed tensors, and 44,544 routed-expert projections. The indexed tensor size is 402,059,722,752 bytes.

The validator checked every routed-expert projection for shape, type, finite positive scales, and index coverage. It checked 174 projections across layers and experts against the expected packed INT4 values bit for bit. The RTN provenance file states that no calibration data was used.

How to run (SGLang)

The same command serves this checkpoint on Blackwell and Hopper. SGLang selects the MoE kernel from the GPU architecture; no checkpoint conversion or backend flag is required. Ampere shares the Marlin code path but was not tested.

python3 -m sglang.launch_server \
  --model-path rakuten-junliu/RakutenAI-3.0-W4A16 \
  --quantization compressed-tensors \
  --kv-cache-dtype bf16 \
  --tp 8 \
  --trust-remote-code \
  --mem-fraction-static 0.85 \
  --port 30000

Wait until curl -sf http://127.0.0.1:30000/health_generate succeeds before sending traffic.

Query the server

curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "あなたは誰ですか?"}],
    "max_tokens": 256,
    "temperature": 0
  }'

With the OpenAI Python client:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Explain mixture-of-experts routing in two sentences."}],
    max_tokens=256,
    temperature=0,
)
print(resp.choices[0].message.content)

Hardware requirements

Stage the checkpoint on node-local storage before TP8 startup when the shared filesystem has limited metadata or read throughput. Every tensor-parallel process scans the checkpoint during load.

B300 output throughput

The test used 1,024 input tokens, 128 output tokens, greedy decoding, a warmup, a radix-cache flush, and one fixed seed per concurrency. The measured launch additionally set --chunked-prefill-size 8192 and --cuda-graph-max-bs-decode 32. All eight ranks captured decode CUDA Graphs for batch sizes through 32.

Known limits

RakutenAI-3.0-W4A8 remains the higher-accuracy internal serving choice. Under the same evaluation protocol and the same pinned client tokenizer, W4A8 scored 73.94 on MMLU-Pro (8,897/12,032) and 79.63 on BBH (5,185/6,511), served on 8×H100. This W4A16 release scored 73.39 and 71.85, served on 8×B300. The two lines come from different GPU generations and MoE kernels; the gap has not been re-measured with both checkpoints on one platform.

The controlled suite covers short-context MMLU-Pro, BBH, and AIME 2024. It does not cover long-context or long-horizon workloads. Evaluate the checkpoint on the target workload before deployment.

License: Apache 2.0, same as the base model.