Laguna-S-2.1-W8A16

Laguna S 2.1 — W8A16 (group_size=32) — tested on 8× RTX 3090

A W8A16 quantization of poolside/Laguna-S-2.1 (118B total / 8B active, 256-expert MoE, OpenMDW-1.1). The high-fidelity companion to Laguna-S-2.1-W4A16 — same model, 8-bit weights for higher quality at the cost of size/speed.

~38 tok/s decode · 120 GB · 256K context (3.18× concurrent) · clean output with thinking on · tested on 8× RTX 3090 (Ampere, 192 GB VRAM)

TL;DR

Why this quant: the high-fidelity variant

The W4A16 is the small/fast option (69 GB, ~92 tok/s). This W8A16 is the higher-fidelity option (8-bit weights preserve more capability — it scored +1 task on Terminal-Bench). Both serve clean on consumer Ampere (3090).

The non-obvious part: group_size=32 is the Laguna-MoE sweet spot for both 4-bit AND 8-bit. The default g128 produces token-artifacts (byfusion, import torchdec) even at 8-bit; g32 clears them — same fix as the 4-bit. And asymmetric 8-bit is kernel-blocked in vLLM (the Marlin WNA16 MoE backend rejects it — is_supported_config → _supports_quant_scheme returns False for asym-8-bit; llm-compressor #2628), so symmetric g32 is the path. The full quant story (OOM wall → per-layer loop → stale-base → group_size → cudagraph) is in the W4A16 blog.

Serve (vLLM)

vLLM 0.25.1+, 8× 24 GB Ampere:

python -m vllm.entrypoints.openai.api_server \
  --model . --served-model-name laguna \
  --tensor-parallel-size 8 --device-ids 0,1,2,3,4,5,6,7 \
  --quantization compressed-tensors \
  --tool-call-parser poolside_v1 --reasoning-parser poolside_v1 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --max-model-len 262144 --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 --trust-remote-code \
  --host 0.0.0.0 --port 8085

Key: --quantization compressed-tensors (required), no --enforce-eager (cudagraphs), --gpu-memory-utilization 0.90, poolside_v1 parsers, enable_thinking.

Terminal-Bench — W8A16 vs W4A16 (same 18 tasks)

W8A16 per-task breakdown:

License & attribution

• Model weights: OpenMDW-1.1 (poolside). Derivative quant — same license.
• Built on an 8×3090 rig (192 GB VRAM) with llmcompressor 0.12 / transformers 5.10.1 / vLLM 0.25.1.