Laguna-XS.2-GGUF

Laguna XS.2 - GGUF Quantizations

First-ever GGUF conversion of poolside/Laguna-XS.2, produced as part of the Poolside Research Hackathon.

Laguna XS.2 is a 33.4B total parameter (3B activated) Mixture-of-Experts model built for long-horizon agentic coding. These GGUF files enable local deployment via llama.cpp and compatible inference engines.

⚠️ Inference status: Full CPU/GPU inference requires 6 C++ patches to llama-model.cpp (documented below). All benchmarks were run using vLLM on the original BF16 checkpoint.

Available Files

Note on quantization sizes: llama.cpp does not quantize 3D MoE expert tensors. Attention/embedding weights are fully quantized; expert weights remain F16. This is a known llama.cpp limitation for large MoE models. Proper small quants (~18GB Q4_K_M) require upstream llama.cpp changes.

Benchmark Results

All benchmarks: H200 MIG 71GB, vLLM 0.21.0, BF16, temperature=0.

MATH500 by Subject

Throughput (BF16, vLLM, H200 71GB)

Benchmark Visualizations

Architecture - Key Differences from Mixtral

Custom GGUF Metadata Keys

C++ Patches for llama.cpp Inference

1. Sigmoid routing: ggml_soft_max → ggml_sigmoid in MoE router
2. Routed scaling: multiply routed output by laguna.moe.routed_scaling_factor (2.5)
3. e_score_correction_bias: add to router scores before top-k
4. Dual RoPE θ: use laguna.rope.theta_swa for SWA layers
5. Partial RoPE: rotate only 50% of head_dim on GA layers
6. Softplus g_proj gating: softplus(g_proj(x)) per head after attention, before o_proj

Conversion

Registration in conversion/__init__.py:

"LagunaForCausalLM": "laguna"

The conversion script (conversion/laguna.py) handles:

• Stacking 256 individual expert weights → 3D tensors [256, dim_in, dim_out]
• Per-layer variable Q-heads (48 GA / 64 SWA)
• Sigmoid router, shared expert, g_proj, e_score_correction_bias
• All Laguna metadata as custom GGUF keys

Original Model

• Model: poolside/Laguna-XS.2
• License: Apache 2.0
• Paper: Technical Report

Produced by Saurabh Mallik for the Poolside Research Hackathon.