Laguna-XS-2.1-W4A16

Laguna-XS-2.1-W4A16

INT4 (W4A16) quantization of poolside/Laguna-XS-2.1 — a ~33B-parameter text Mixture-of-Experts causal LM (LagunaForCausalLM, custom code) from poolside: 40 layers (dense MLP at layer 0, sparse MoE with 256 routed experts + a shared expert, 8 experts/token, in layers 1-39), alternating full/sliding attention with a per-head output gate, YaRN RoPE, and a 262K context.

Variant: W4A16 — 4-bit symmetric integer weights, group size 128, activations BF16. Round-to-nearest (data-free, no activation-aware scaling). Quantized by: sahilchachra Tooling: llm-compressor model_free_ptq (data-free, RTN) -> compressed-tensors

This is a quantized derivative. Weights, behavior, and license follow the base model — see the original card for full details, benchmarks, and citation.

What is quantized

Quantized to 4-bit:

• routed experts mlp.experts.*.{gate,up,down}_proj (layers 1-39)
• shared expert mlp.shared_expert.{gate,up,down}_proj
• dense MLP mlp.{gate,up,down}_proj (layer 0)
• attention self_attn.{q,k,v,o}_proj (all layers)

Kept in BF16: MoE router mlp.gate, attention output gate self_attn.g_proj, router e_score_correction_bias, token embeddings, lm_head, all norms (incl. q_norm / k_norm).

Calibration

Data-free — weight-only (model_free_ptq, round-to-nearest); no calibration data. Weights are quantized by streaming the safetensors from disk.

Usage (vLLM)

from vllm import LLM, SamplingParams

# Weight-only quantized (custom architecture -> requires trust_remote_code).
# Load like the original model in any runtime that implements this arch.
llm = LLM(
    model="sahilchachra/Laguna-XS-2.1-W4A16",
    trust_remote_code=True,
)
out = llm.chat(
    [{"role": "user", "content": "Hello!"}],
    SamplingParams(temperature=0.6, top_p=0.95, max_tokens=512),
)
print(out[0].outputs[0].text)

Serving via the CLI, pass the flag directly:

vllm serve sahilchachra/Laguna-XS-2.1-W4A16 \
    --trust-remote-code \
    --max-model-len 262144