Laguna-S-2.1-Selective-AWQ

Laguna S 2.1 Selective AWQ — W4A16 g32

Checkpoint size: 94.30 GB (87.82 GiB) of safetensors weights.

This project was built to make the 118B-parameter Laguna S 2.1 MoE model usable on 4× Tesla V100 32GB with tensor parallel size 4 and a checkpoint below 100 GB. The size ceiling, W4A16 g32 asymmetric quantization, FP16 weight storage, and the protected routed-expert allocation—Layers 1–4 and 46–47—were all selected for that deployment target.

The main checkpoint is the general form of the same work: one unscaled weight artifact configured for BF16 activations, intended to run directly on a BF16-capable SM80+ system with stock vLLM. The V100 runtime add-on reuses those exact weights; there is no second V100 checkpoint.

This is an independently produced selective AWQ quantization of poolside/Laguna-S-2.1. It is not an official Poolside release.

Critical dtype requirement for the main checkpoint

Run the unscaled main checkpoint with BF16 activations. Do not force FP16 (float16/half).

The original model can produce very large late-layer hidden states at BOS/EOS anchor positions. In our fixed calibration replay, the largest observed BF16 hidden value was approximately 950,272, above the FP16 finite limit of 65,504. Each of the 1,097 token-id-2 BOS/EOS anchor rows contained at least one component outside the FP16 range, while the largest observed non-anchor value was 17,152. Normalized outputs and logits remained finite in BF16.

This is an activation dynamic-range property, not an AWQ packing failure. A runtime that silently executes the main checkpoint with FP16 activations can produce non-finite logits at EOS positions.

V100 runtime add-on — primary deployment target

The included v100-1cat/ directory is a small, fail-closed runtime patch for the configuration that motivated this quantization:

• 4× Tesla V100 PCIe 32GB;
• tensor parallel size 4;
• 1Cat-vLLM 1.2.2;
• FP16 activations and FP16 KV cache;
• TRITON_ATTN, chunked prefill, and prefix caching disabled.

It is deliberately version-pinned, not a cross-version plugin. It uses the exact main-model revision without copying or permanently rewriting the approximately 94 GB checkpoint.

Rc4 fixes two independent FP16 runtime problems. The pinned 1Cat/vLLM MoE runner's overflow-protection branch represents Laguna's combined routed/shared output at 1 / routed_scaling_factor and requires a matching decoder compensation. The pinned Laguna decoder omitted that multiply, so each MoE layer returned approximately 0.4x of the intended branch output for the configured factor 2.5. Rc4 restores the required FP16 x2.5 after every MoE layer. BF16 inference does not use this branch and is unaffected.

The second problem is the late-layer BOS/EOS activation overflow. For that range repair, the add-on applies a local power-of-two reparameterization to Layers 45–47, while the forward path multiplies hidden/residual by 1/64 once before Layer 45:

• attention o_proj: × 1/64;
• routed/shared expert up_proj: × 1/256;
• routed/shared expert down_proj: × 4.

The net branch factor is 1/64. Layer 45 remains W4; its qweight and qzeros are unchanged, while the relevant floating AWQ scale carriers are transformed. The add-on rejects an unknown model revision, quantization layout, or tensor count.

FLASH_ATTN_V100 is deliberately rejected by the add-on. On this model, the tested Flash-V100 path became non-finite after chunked prefill crossed its observed boundary. Triton is somewhat slower on short prompts, but it is the accepted V100 path from 512 tokens through long context.

The same-weight loader-time implementation has reproduced the materialized reference:

The non-zero KLD and 99.9391% top-1 figures compare rc4's Layer-45 /64 repair with an unscaled FP16 runtime that includes the same required MoE x2.5 correction. The diagnostic PPL changed from 21.9363 to 21.9160, but 1Cat's Laguna prompt-logprob token attachment is shifted by one position, so that value is not presented as a language-model perplexity result.

The 30-case smoke used thinking=off with a 512-token cap. It is a small gross-regression check, not a public benchmark or an exact numerical gate.

A standalone global /32 weight-folding experiment was also tested and rejected: against the same unscaled baseline it produced 0.0661285 nats mean coarsened KLD and 91.0010% token top-1 agreement because FP16 rounding accumulated through all 48 layers. The V100 release therefore uses the local runtime add-on.

Model summary

Laguna S 2.1 is a text-to-text, 118B-total-parameter, approximately 8B-active-per-token Mixture-of-Experts model for agentic coding and long-horizon work. It is not a vision or audio model. The base model has 48 decoder layers, 256 routed experts plus one shared expert, and a configured context window of 1,048,576 tokens. See the official base model card for architecture, chat formatting, intended use, responsible-use guidance, and base-model benchmark results.

Frozen base revision:

poolside/Laguna-S-2.1@e80da38da3ed4c4e56888cc1ba39582946a164ba

Quantization layout

Layer numbers are zero-based Hugging Face names, model.layers.N.

AWQ configuration:

{
  "bits": 4,
  "group_size": 32,
  "quant_method": "awq",
  "version": "gemm",
  "zero_point": true
}

W4A16 denotes packed 4-bit weights with 16-bit activations. The main path uses BF16 activations; the V100 add-on uses FP16 activations with the local range repair above.

A single FP16-stored weight artifact serves both the BF16 main path and the V100 add-on. The precision effect of storing protected tensors in FP16 is already included in the KLD, perplexity, and top-1 measurements reported below.

Some protected FP16 tensors participate in mathematically equivalent AWQ scale compensation. They remain FP16 and were not packed to W4, but they need not be byte-identical to the corresponding base-model tensors.

quantization_config.modules_to_not_convert has been checked against this exact FP16/W4 layout and the machine-readable tensor audit. Stock vLLM 0.24.0 also loaded the mixed layout successfully; this is no longer inferred only from the JSON declaration.

Why this allocation?

The protected prefix and tail were selected from actual packed-model comparisons under the sub-100-GB V100 budget.

The selected layout protects four early and two final routed-expert layers while remaining below the 100,000,000,000-byte checkpoint limit. The 30-task column is a small deterministic regression smoke test, not a general benchmark.

Main-checkpoint usage

The supported main path uses upstream vLLM's native Laguna implementation and does not depend on copied model remote code. Transformers inference is not currently a validated path.

The command below is the intended BF16 serving form. The Poolside parser flags remain separately qualified below because the stock-vLLM probe exercised the native model and dtype path, not the HTTP parser stack:

vllm serve \
  --model leoncca/Laguna-S-2.1-Selective-AWQ \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --tool-call-parser poolside_v1 \
  --reasoning-parser poolside_v1 \
  --enable-auto-tool-choice \
  --served-model-name laguna \
  --default-chat-template-kwargs '{"enable_thinking": true}'

The parser and chat-template flags are copied from the base model's recommended vLLM setup. Reasoning and forced tool calls passed on the validated 1Cat V100 runtime; the same parser flags have not yet been re-run through the upstream-vLLM HTTP server.

The accepted stock-vLLM probe positively verified the loaded dtype rather than relying on the absence of a cast warning. On both TP workers, the 917 exposed parameter tensors consisted of 624 BF16, 47 FP32 router-bias, and 246 INT32 packed tensors, with no FP16 parameter. Hooks observed all 48 decoder layers with BF16 finite hidden/residual tensors and a finite BF16 final norm.

Upstream vLLM 0.24.0 contains a native Laguna model implementation and its AutoAWQ configuration accepts BF16 activations. The exact RC was loaded with trust_remote_code=False, dtype=bfloat16, and quantization=auto_awq.

Compatibility status

The library_name: transformers metadata describes the Hugging Face configuration/tokenizer format and base architecture; it is not a claim that Transformers inference has been validated.

Quantization and evaluation data

Calibration

Calibration data was used for post-training quantization only; the base model was not retrained.

Held-out numerical comparison

The main numerical comparison used 16 disjoint 512-token held-out blocks and 8,176 scored next-token positions against the fixed raw-BF16 base revision. Decoder replay used BF16, final normalization accumulated in FP32, and the protected FP16 lm_head used the FP16 logit-compute path.

Held-out token tensor SHA-256:

77976641882fa641e26f998594aa935518acb7796e9907e7182656b1361015ce

These measurements describe quantization drift on a fixed corpus. They do not claim to reproduce the base model's published software-engineering benchmark scores.

Artifact integrity

Frozen release checkpoint:

Checkpoint shard bytes counts only the 49 safetensors files. The shards and model index remain frozen. The release process regenerates the repository-root SHA256SUMS for the corrected rc4 add-on. The public file set and its integrity metadata were verified against the frozen rc4 candidate before the v1.0 tag was created. Additional clean-download runtime checks are tracked separately and are not a prerequisite for reading or verifying this checkpoint.

Context length

The checkpoint inherits the base model's configured maximum context of 1,048,576 tokens, but that length has not been validated for this exact quantized release. Rc4 passed 4,095-, 15,352-, and 259,989-token needle tests and exposes 395,822 KV tokens at the validated 262,144-token server profile. The 259,989-token run recovered the exact needle with 1,384.602 s TTFT (187.77 prompt tok/s); peak sampled memory was 31,968 MiB used and 527 MiB free on each V100. One-million-token context remains not evaluated.

Intended use and limitations

This derivative is intended for local or hosted text generation and agentic coding workflows where the selective W4/FP16 trade-off is useful, particularly 4× V100 32GB TP4 deployments.

• The main checkpoint requires BF16 activations; forcing FP16 can overflow at EOS positions.
• The V100 path requires the version-pinned add-on and must not be assumed compatible with other runtimes.
• Only routed experts in Layers 5–45 are W4, so this is larger than a uniform all-expert W4 checkpoint by design.
• The stock-vLLM BF16 compatibility test used heterogeneous TP2 plus 36 GiB CPU offload per GPU at a 512-token profile. It validates native loading and dtype/range behavior, not production speed or long-context capacity on that machine.
• Transformers/AutoAWQ, SGLang, TensorRT-LLM, standalone RTX 3090, DFlash, MTP, and 1M context are not evaluated.
• Quantization can change generations even when aggregate numerical metrics remain close.

Follow the original model card's chat-template, thinking-preservation, intended-use, and responsible-use guidance.

License and attribution

The original model is poolside/Laguna-S-2.1, created by Poolside and released under OpenMDW-1.1. This repository retains the complete original LICENSE.md, copyright notices, and source notices required by that license.

Quantization and packaging: leon

Original model: poolside/Laguna-S-2.1

Original license: OpenMDW-1.1

The V100 runtime patch retains the applicable upstream copyright notices and Apache-2.0 license in v100-1cat/.

Release information

• Hugging Face publication target: leoncca/Laguna-S-2.1-Selective-AWQ
• Release tag: v1.0 (created from the remotely verified release revision)
• Publisher/quantizer: leon
• Main BF16 validation hardware/runtime: upstream vLLM 0.24.0; RTX 3090 24GB (SM86) + RTX 5060 Ti 16GB (SM120), TP2, 36 GiB CPU offload/GPU; PyTorch 2.11.0+cu130; Transformers 5.14.1; native LagunaForCausalLM; trust_remote_code=False
• V100 patched Laguna runtime SHA-256: 144ca4515847be808c12ca06a9d8c563a6be8a15d0c90953c16cfd30c5667fa4
• Integrity file: repository-root SHA256SUMS; the public file set and integrity metadata were checked against the frozen rc4 candidate before v1.0 was created. Additional clean-download runtime checks remain tracked separately.