Laguna-S-2.1-NVFP4

Laguna-S-2.1-NVFP4

A faster, leaner build of poolside/Laguna-S-2.1 for local Blackwell — up to 1.67× faster decode and ~4 GB lower VRAM, at measured quality parity. Drop-in for vLLM.

Poolside built Laguna-S-2.1 as an agentic software-engineering model — and for its size (118B total, ~8B active MoE) it's a capable coder. But it isn't narrow: it keeps broad general knowledge and reasoning (MMLU 0.81, GSM8K 0.92), and its native tool-calling and multi-step behavior make it a strong general-purpose agentic model — well-suited to non-coding work like support and operations automation, research assistants, and multi-step coordination, not just code.

Highlights

• Faster single-stream decode across Blackwell — 1.67× on a DGX Spark (GB10, 32.0 vs 19.1 tok/s) and 1.38× on an RTX Pro 6000 (145.7 vs 105.5 tok/s), wherever decode is memory-bandwidth-bound.
• ~4 GB smaller weights — the attention quant cuts the model's weight footprint by ~4 GB (device-resident and on-disk), freeing memory for more context and larger batches on any Blackwell GPU.
• Quality parity within measured evaluation variance — matches the stock NVFP4 checkpoint across MMLU, HellaSwag, GPQA-Diamond, GSM8K, and agentic SWE-bench (all deltas sub-σ).
• Scales from one Spark to two — a second GB10 (TP=2 over 400G) adds ~1.7× concurrent-serving throughput.
• Drop-in — quantization is auto-detected; identical serving recipe to poolside/Laguna-S-2.1-NVFP4, including DFlash speculative decoding and the 256K→1M context switch.

Accuracy & quality

Measured against the stock poolside/Laguna-S-2.1-NVFP4 as the reference, on a neutral serve (no sampling overrides). The multiple-choice suites use loglikelihood; GSM8K and SWE-bench are generative.

* SWE-bench Multilingual is single-attempt agentic (n=32 like-for-like). The −1-instance difference is ≈0.4σ — inside normal agentic-loop run-to-run variance, and consistent with the zero-degradation measured on the large-n multiple-choice suites. It is not a measurable regression; the agentic test is the most sensitive to attention quantization, which is why it is reported here.

<details> <summary><b>Evaluation methodology</b></summary>

Both this build and the stock reference were evaluated on the same vLLM 0.25.1 / RTX Pro 6000 (sm_120) stack — compressed-tensors NVFP4 kernels (MarlinNvFp4 GEMM + FlashInfer-CUTLASS NVFP4 MoE), greedy (temperature 0), neutral serve (no sampling overrides), --kv-cache-dtype fp8. NVFP4 numerics are consistent across the Blackwell backends, so these figures transfer to the GB10/Marlin serving path.

• Multiple-choice — lm-evaluation-harness v0.4.12. MMLU (5-shot), HellaSwag, GPQA-Diamond via loglikelihood (--model local-completions, no chat template); GSM8K (5-shot) generative (local-chat-completions, chat template applied).
• SWE-bench Multilingual (test split) — mini-SWE-agent 2.4.6 + SWE-bench harness 4.1.0, native tool-calling scaffold (poolside_v1 reasoning/tool parsers, auto tool choice), 250-step limit, thinking off, single attempt, 8 workers. Agentic numbers are scaffold-dependent; this is the scaffold used. n=41 raw (this build 27/41, stock 30/41); like-for-like on the 32 instances where both produced a patch: 25/32 vs 26/32.

</details>

Performance

All figures are measured on vLLM 0.25.1, temperature 0, --kv-cache-dtype fp8, at tokenizer-exact context lengths, single stream.

Decode on a single DGX Spark (GB10), single-stream (2K):

The fastest single-GB10 config is this build + DFlash, 43.2 tok/s. DFlash speeds up both builds on one node; the stock model gains more in relative terms (it starts slower) but never overtakes this build in a like-for-like config.

Two Sparks (TP=2, 2× ConnectX-7 = 400G).

A second GB10 raises single-stream decode to 46.0 tok/s (1.60× stock's 28.8) — just past a single node with DFlash (43.2) — and adds concurrent-serving throughput. Serve on one GB10 for the lowest-cost single stream; add a second for higher single-stream latency and concurrency.

⚠️ On two nodes, run this build without spec-decode. Even with the DFlash draft pinned to a single node (draft_tensor_parallel_size=1, so it pays no cross-node cost — verified from the serve logs), DFlash reaches only 40.6 tok/s, below plain no-spec's 46.0 — because this build's quantized attention already makes no-spec decode bandwidth-fast, leaving nothing for speculation to recover. (DFlash does help the stock model on two nodes, 28.8→39.2, which starts from a slower baseline.) Expert-parallel (--enable-expert-parallel) is a wash vs TP=2 here. Use DFlash on a single GB10; run plain on two.

Decode vs context length (single GB10) — the advantage holds as context grows, because Laguna's sliding-window attention keeps KV growth small (36 of 48 layers use a 512-token window):

Prefill throughput scales with context as chunked prefill amortizes — this build reaches 25K / 145K / 290K tok/s at 2K / 16K / 64K on one GB10, matching or ahead of stock. DFlash lifts decode but adds to TTFT, roughly halving prefill throughput, so prefer it for decode-heavy (long-generation) workloads.

RTX Pro 6000 (sm_120, discrete Blackwell): single-stream decode 145.7 vs 105.5 tok/s = 1.38× faster — RTX is memory-bandwidth-bound at batch 1, so quantizing attention speeds it up here too. At 32-way concurrency the workload turns compute-bound (NVFP4 MoE GEMMs dominate) and the two builds converge (1399 vs 1315 tok/s, 1.06×), so the RTX gain is an interactive/single-stream one. The ~4 GB smaller footprint applies regardless.

Tuning DFlash (single GB10). Draft depth k=3 is optimal; higher k loses because each extra verified token pulls in more MoE experts (the verify is a mini-prefill) while acceptance saturates (~2.85 accepted tokens/cycle). Poolside's example k=15 is tuned for B200-class bandwidth, where the expert-read barely registers.

Concurrent serving (two Sparks). For many parallel users the best layout flips. Pure expert-parallel — --data-parallel-size 2 --enable-expert-parallel (one 128-expert set per node, --tensor-parallel-size 1, no TP) — overtakes TP=2 as concurrency rises, reaching 790 vs 505 tok/s at 128-way (1.56×); TP=2 still wins single-stream. Rule of thumb: TP=2 for latency, DP+EP for throughput. (Spec-decode is a single-stream tool — it collapses under concurrency on TP=2, so leave it off for high-throughput serving.)

Running on a DGX Spark

Two ways to run on a single GB10, depending on how much single-stream speed you want:

Both serve an OpenAI-compatible endpoint. Start with A; use B for the fastest single stream (it bundles our 4-bit DFlash drafter, which has a cheaper draft step, plus the one loader fix vLLM still needs to load a 4-bit drafter).

A. Standard — vLLM drop-in

1. One-time environment (vLLM 0.25.1 + FlashInfer for sm_121):

sudo apt install -y python3.12-dev                       # Triton JIT needs the headers
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv ~/venvs/vllm025 -p 3.12
uv pip install -p ~/venvs/vllm025 vllm==0.25.1 --torch-backend=cu130
uv pip install -p ~/venvs/vllm025 \
  "flashinfer-python==0.6.15.dev20260712" "flashinfer-cubin==0.6.15.dev20260712" "flashinfer-jit-cache==0.6.15.dev20260712" \
  --extra-index-url https://flashinfer.ai/whl/nightly/ --extra-index-url https://flashinfer.ai/whl/nightly/cu130/ \
  --index-strategy unsafe-best-match
source ~/venvs/vllm025/bin/activate
export CUTE_DSL_ARCH=sm_121a MAX_JOBS=4                   # FP4 kernel JIT arch; cap nvcc fan-out

2. Serve (weights auto-download from this repo; quantization is auto-detected). This includes DFlash speculative decoding with poolside's matched BF16 drafter:

vllm serve ennerd/Laguna-S-2.1-NVFP4 \
  --trust-remote-code --kv-cache-dtype fp8 --moe-backend marlin \
  --reasoning-parser poolside_v1 --tool-call-parser poolside_v1 --enable-auto-tool-choice \
  --override-generation-config '{"temperature":0.7,"top_p":0.95}' \
  --speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":3}' \
  --max-num-seqs 32

→ ~44 tok/s single-stream. Drop the last two lines for a plain ~32 tok/s serve without speculation. (--max-num-seqs 32 is required with DFlash — it errors at the default of 256.)

B. Maximum performance — Docker with our bundled 4-bit drafter

One docker build + docker run gives you the fastest single-stream on a Spark, with no venv/FlashInfer setup and no manual patching — the image bakes in vLLM 0.25.1, the loader fix, and our 4-bit DFlash drafter (in dflash/).

1. Get the repo (the Dockerfile and the dflash/ drafter live here):

hf download ennerd/Laguna-S-2.1-NVFP4 --local-dir laguna && cd laguna

2. Build the image:

docker build -t ennerd-laguna-spark .

3. Run it (serves on :8000; mounts the weights you just downloaded):

docker run --gpus all --ipc host -p 8000:8000 -v "$PWD":/model ennerd-laguna-spark

→ ~47.7 tok/s single-stream. Then query it:

curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"Laguna-S-2.1","messages":[{"role":"user","content":"Write quicksort in Python."}]}'

Draft depth k=3 is the measured GB10 optimum (set NUM_SPEC to change it); see DOCKER.md for all knobs. Run this on one GB10 — on two Sparks, serve without speculation (see Performance).

Context: 256K and 1M

These are native 1M-trained weights, shipping defaulted to a 256K window. To restore 1M, apply the base model's published 1M RoPE settings (these constants come from poolside/Laguna-S-2.1-NVFP4's 1M configuration — they are not arbitrary) to config.json:

"rope_parameters": { "full_attention": { "factor": 128.0, "attention_factor": 1.4852030263919618 } },
"max_position_embeddings": 1048576

How it works

Relative to the stock poolside/Laguna-S-2.1-NVFP4, this build additionally quantizes the attention projections (q_proj, k_proj, v_proj, o_proj) to NVFP4, weight-only. The routed experts, lm_head, shared MLP, and norms are unchanged from the base checkpoint.

The stock NVFP4 checkpoint keeps attention, lm_head, and the shared MLP in BF16 — the right trade on B200-class bandwidth (~8 TB/s). But local Blackwell decode is memory-bandwidth-bound: at batch 1 the weight read dominates, and on a DGX Spark (GB10, 273 GB/s unified memory) that BF16 attention is ~57% of per-token weight traffic. Quantizing it too cuts per-token weight traffic from ~9.9 GB to ~5.7 GB — and because decode time scales with bytes-read, that is the decode speedup. The win holds anywhere decode is bandwidth-bound: unified-memory GB10, and concurrent serving on any Blackwell (batching amortizes compute while the weight read stays fixed).

Limitations. This is a weight-only quant — it reduces memory traffic and footprint, so it accelerates bandwidth-bound decode but adds no activation-quantization benefit for compute-bound serving. On a high-bandwidth GPU at large batch (e.g. RTX Pro 6000 at 32-way concurrency), the workload is compute-bound and this build converges with stock NVFP4 on speed; there the gain is the ~4 GB smaller footprint, not throughput.

<details> <summary><b>Compatibility & kernel notes</b></summary>

• Runs packed — the model weights stay 4-bit at load (~67 GB of packed weights — device-resident and on-disk, excluding KV cache and activations), not decompressed to BF16 — on both Blackwell targets: GB10 (Marlin NVFP4) and sm_120 discrete Blackwell (MarlinNvFp4 attention + FlashInfer-CUTLASS NVFP4 MoE).
• sm_120 + CUDA < 12.9: the FlashInfer-CUTLASS NVFP4 MoE kernel wants CUDA ≥ 12.9. On CUDA 12.8, add --moe-backend marlin (CUDA-version-tolerant, numerically identical).
• Serve via vLLM / SGLang / TensorRT-LLM, not plain Transformers — Laguna's fused 3-D expert parameters can't be kept packed by the Transformers quant path, which decompresses the experts to BF16 at load (~235 GB).

</details>

Reproduction

The attention projections (q_proj, k_proj, v_proj, o_proj) are quantized to NVFP4 weight-only and data-free: per-16-element block scales (FP8-E4M3) plus a per-tensor FP32 global scale, derived from the weights themselves (min/max) — no calibration dataset. The quantization uses compressed-tensors with the same quantization arguments as the base checkpoint's own routed experts. Before touching attention, we validated that our quantizer reproduces those base experts bit-exactly (0.000 relative error), so the method is provably identical to Poolside's. The routed experts, lm_head, shared MLP, and norms are byte-identical to the base checkpoint; only attention is added to the quantized set.

Source checkpoint: poolside/Laguna-S-2.1-NVFP4 @ commit 0761412.

Attribution & license

• Base model: poolside/Laguna-S-2.1 — © poolside. This is a quantized derivative of poolside/Laguna-S-2.1-NVFP4.
• License: OpenMDW-1.1, unchanged from the base model. This redistribution includes the OpenMDW-1.1 agreement and preserves the original copyright and origin notices.
• Responsible use: a software-engineering and agentic-coding model; confirm suitability for your application and use it consistently with Poolside's Acceptable Use Policy. Report security or safety concerns to [email protected].

---

Published by Ennerd. Not affiliated with or endorsed by poolside.