legunas21-custom

Laguna-S-2.1 on AMD Strix Halo (llama.cpp + Vulkan) — deployment kit

Working configuration, tuning notes, and benchmark data for running poolside/Laguna-S-2.1 (118B MoE, ~8B active) on an AMD Strix Halo iGPU (Radeon 8060S, RDNA3.5, unified memory) under OpenShift via the llama.cpp Vulkan backend (RADV).

Final serving stack: Myric APEX i-compact quant (54.4 GB, MoE-aware mixed precision) + metadata-fixed DFlash Q4_K_M draft, on poolside's llama.cpp fork (branch laguna).

Hardware: 16 GiB VRAM carve-out + ~55 GiB GTT on a 114 GiB node.

Files

Removed: earlier revisions of this repo mirrored community Q4/Q8 DFlash conversions made from poolside's pre-fix upload. Those files are missing the sliding-window metadata and silently collapse to <1% draft acceptance beyond a few K tokens of context (8 t/s at 45K on our hardware). Check git history if you need them; you don't.

Benchmarks (2026-07-23/24, Strix Halo, 114 GiB, Vulkan/RADV)

Shallow = 512-token code gen. Deep = 45K-token prompt, 256-token continuation.

Deep-context decode improved 2.3× over the original deployment. The DFlash speedup grows with depth: the target slows (KV reads over 45K tokens) while the SWA-corrected draft's cost stays constant (512-token window), so drafting gets relatively cheaper exactly when you need it.

Looping smoke test on APEX i-compact: 3 tool-less diagnostic prompts + 1 adversarial physics puzzle — all terminated with finish_reason=stop, no unclosed thinking blocks. The model still overthinks hard problems (~11K reasoning tokens on the physics puzzle) — that is a model trait at any precision; give it max_tokens ≥ 8k and consider a "think briefly, then act" system prompt.

The DFlash draft metadata bug (why your acceptance collapses at depth)

Poolside's day-one DFlash GGUF lacked sliding-window attention metadata; they fixed it in discussion #12 ("Correct DFlash config", 2026-07-22). Community Q4/Q8 conversions made from the pre-fix file inherit the bug: fine at shallow depth, useless past ~2-4K tokens. Diagnose by checking your draft GGUF for dflash.attention.sliding_window = 512; fix by requantizing the current official BF16:

llama-quantize laguna-s-2.1-DFlash-BF16.gguf laguna-s-2.1-DFlash-Q4_K_M-fixed.gguf Q4_K_M 8

(Quantization preserves metadata — or just use the fixed file from this repo.)

Serving command (poolside fork, branch laguna)

llama-server \
  -m Laguna-S-2.1-APEX-i-compact.gguf \
  -md laguna-s-2.1-DFlash-Q4_K_M-fixed.gguf \
  --spec-type draft-dflash --spec-draft-n-max 4 \
  -ctkd f16 -ctvd f16 -ngld 99 \
  --temp 0.7 --top-p 0.95 \
  -c 65536 -ngl 99 -fa on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --jinja --parallel 1

Hard-won details

1. DFlash needs poolside's llama.cpp fork (poolsideai/llama.cpp, branch laguna). Upstream (b10087–b10103 tested) fails on any DFlash draft with expected 76, got 69 — the draft borrows token_embd/output.weight from the target. Apply speculative-cmath.patch for GCC 15 / Ubuntu 26.04.
2. Use the SWA-fixed draft (see above) and draft KV in f16 (-ctkd f16 -ctvd f16) — quantized draft KV also collapses acceptance.
3. --spec-draft-n-max 4 measured best on bandwidth-limited hardware (poolside's 15, and even their updated 7, were net losses here with a Q4 draft).
4. unsloth GGUF metadata bug (if serving unsloth quants): they ship laguna.rope.scaling.yarn_attn_factor = 1.4852, double-applied by llama.cpp. Fix: --override-kv laguna.rope.scaling.yarn_attn_factor=float:1.0. The APEX quants don't need this.
5. Sampling: the model card's --temp 0.7 --top-p 0.95 (the shipped generation_config's 1.0/1.0 measurably increases fabrications). No min-p.
6. Flash attention ON: 2.6× decode / 1.7× prefill vs off on RDNA3.5 Vulkan. The RDNA3.5 FA crash reports were ROCm-path; Vulkan/RADV ran clean.
7. Vulkan env for Strix Halo: AMD_VULKAN_ICD=RADV, VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/radeon_icd.json, HIP/ROCR disabled.
8. Quant choice on unified memory: decode speed tracks file size almost linearly (bandwidth-bound). APEX's MoE-aware layout (shared expert Q6_K, attention Q4_K, middle routed experts compressed hard) kept quality while dropping 3 GB vs UD-IQ4_XS — and cured nothing that wasn't already fine, since looping was a pre-fix-GGUF/2-bit-quant phenomenon, not a 4-bit UD one.

License

OpenMDW-1.1, inherited from Laguna S 2.1. The fixed DFlash draft is a direct Q4_K_M requantization of poolside's official DFlash BF16 GGUF (current, post-fix revision); poolside does not endorse this redistribution.