Laguna-S-2.1-W4A16

Laguna S 2.1 — W4A16 (group_size=32) — tested on 8× RTX 3090

A W4A16 quantization of poolside/Laguna-S-2.1 (118B total / 8B active, 256-expert MoE, OpenMDW-1.1). Same model, compressed to fit consumer Ampere — see the parent for the full model card, capabilities, and license.

A clean W4A16 quantization of poolside/Laguna-S-2.1 (118B total / 8B active, 256-expert MoE) that serves fast on consumer Ampere GPUs — the cards with no native FP8/FP4. No official W4A16 for Ampere exists (poolside ships QAT-INT4 that loads poorly on Ampere vLLM, and FP8/NVFP4 which are Blackwell-only). This fills that gap.

~92 tok/s decode · 69 GB · 256K context (7.33× concurrent) · clean output with thinking on · tested on 8× RTX 3090 (Ampere, 192 GB VRAM)

TL;DR

Why this quant exists

Laguna S 2.1 is a top open agentic-coding MoE that fits consumer hardware — but only if you can quantize it without OOMing, pick the right group_size, and serve it without leaving throughput on the floor. The full story (the llmcompressor OOM wall, the per-layer fix, the stale-base red herring, the group_size correction, the cudagraph win) is in LAGUNA-W4A16-BLOG.md.

The short version of the two non-obvious wins:

1. group_size=32, not 128. The Qwen3.5-style default group_size=128 is too coarse for Laguna's 256-expert MoE — large per-group error flips logits at token boundaries and splices stray number/special tokens into technical + markdown output. Poolside's own reference INT4 uses g32. At g32 the corruption disappears. (Set via QuantizationModifier's config_groups, not scheme — the scheme field only accepts preset names.)
2. CUDA graphs on (--enforce-eager removed). With ~8B active params, per-step Python + MoE-routing-dispatch overhead dominates the tiny compute in eager mode. Cudagraphs fuse it out: ~7 tok/s → ~92 tok/s (12.5×) at the cost of a hair less KV (7.65× → 7.33× concurrency @ 256K).

Serve (vLLM)

vLLM 0.25.1+, 8× 24 GB Ampere. The cudagraph config is the whole game:

python -m vllm.entrypoints.openai.api_server \
  --model . --served-model-name laguna \
  --tensor-parallel-size 8 --device-ids 0,1,2,3,4,5,6,7 \
  --quantization compressed-tensors \
  --tool-call-parser poolside_v1 --reasoning-parser poolside_v1 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --max-model-len 262144 --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 --trust-remote-code \
  --host 0.0.0.0 --port 8085

Key flags: --quantization compressed-tensors (required), no --enforce-eager (cudagraph = the 12× lever), --gpu-memory-utilization 0.90 (higher over-reserves KV and OOMs sampler warmup), poolside_v1 tool/reasoning parsers, enable_thinking. Full runnable script: Laguna-S-2.1-W4A16-vLLM.sh.

Terminal-Bench results

18-task random subset of terminal-bench-core==0.1.1, terminus agent, pass@1, default turn budget, 900s agent cap. Honest sample — not the full suite.

Notable: several fails were near-solves (e.g. one task passed 8/9 sub-tests), and the hard infrastructure tasks (QEMU / kernel-build / maze / video-extraction) hit the agent timeout — expected for an 8B-active model on a frontier-calibrated benchmark (the TB paper has frontier models <65% on the full suite).

Sample agent reasoning trace

Laguna drives the terminus agent with structured per-episode reasoning. On the hf-model-inference task (build a Flask sentiment API around a HF model), episode 1:

state_analysis: "Packages installed successfully (torch, transformers, flask). Now need to download the model to /app/model_cache/sentiment_model and create the Flask API." explanation: "Will download the Hugging Face model using the transformers library and save it to the specified cache directory. Then create the Flask API application." commands: mkdir -p /app/model_cache/sentiment_model → python3 -c "from transformers import AutoTokenizer, AutoModelForSequenceClassification; … model.save_pretrained('/app/model_cache/sentiment_model')"

Concrete evidence the quant retains agentic reasoning, not just chat fluency.

License & attribution

• Model weights: OpenMDW-1.1 (poolside). Derivative quant — same license.
• Quant scripts + recipe + blog: MIT-adjacent, use freely.
• Built on an 8×3090 rig (192 GB VRAM, ~251 GB RAM) with llmcompressor 0.12 / transformers 5.10.1 / vLLM 0.25.1.

Files